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14.  ABSTRACT 

The  UH-60M  Product  Office  requested  the  U.S.  Army  Research  Laboratory’s  (ARL’s)  Human  Research  and  Engineering  Directorate  to 
participate  in  the  Limited  Early  User  Evaluation  (LEUE)  of  the  Common  Avionics  Architecture  System  (CAAS)  cockpit.  ARL  conducted  a 
human  factors  evaluation  (HFE)  during  the  LEUE,  which  assessed  workload,  situation  awareness,  simulator  sickness,  pilot- vehicle  interface 
(PVI),  and  eye  tracker  data.  The  data  were  used  to  identify  characteristics  of  the  CAAS  cockpit  that  enhance  or  degrade  pilot  performance. 
Characteristics  that  degrade  pilot  performance  should  be  considered  for  design  changes  at  the  earliest  opportunity. 

Three  utility  helicopter  (UH)-60  crews  (six  pilots)  each  conducted  three  mission  scenarios  for  a  total  of  nine  flights.  The  three  missions 
consisted  of  flights  in  visual  meteorological  conditions  (VMC),  instrumented  meteorological  conditions  (IMC),  and  tactical  conditions.  The 
pilots  completed  the  simulator  sickness  questionnaire  before  and  after  each  flight.  They  completed  the  Bedford  Workload  Rating  Scale, 
Situation  Awareness  (SA)  Rating  Technique,  and  the  PVI  Questionnaire  after  each  mission.  In  addition  to  pilot  data,  a  tactical  steering 
committee  (TSC)  was  used  to  perform  an  independent  assessment  of  workload,  situation  awareness,  and  mission  success.  The  TSC  completed 
a  survey  after  each  mission.  The  data  were  analyzed  with  the  use  of  the  Wilcoxon  Signed  Ranks  Test  to  compare  pilot  ratings  between  seat 
position  and  results  between  instrument  flight  rule  (IFR)  and  visual  flight  rule  (VFR)  flights. 

The  mean  workload  rating  for  all  tasks  was  3.10,  indicating  that  the  pilots  typically  had  enough  workload  capacity  for  all  desirable  additional 
tasks.  The  mean  situation  awareness  rating  provided  by  the  pilots  was  25.84.  This  SA  rating  indicates  that  the  pilots  felt  they  had  moderate 
levels  of  situation  awareness  during  the  missions.  The  pilots  also  provided  a  lot  of  data  and  comments  regarding  the  PVI  and  offered 
recommendations  for  design  improvements.  Finally,  the  eye  tracker  results  showed  that  the  flying  pilot  was  focused  out  the  window  60.86% 
of  the  time,  while  the  non-flying  pilot  spent  only  26.30%  focused  out  the  window. 

The  results  of  the  assessment  concluded  that  the  CAAS  cockpit  resulted  in  acceptable  workload  and  situation  awareness  levels;  however, 
several  issues  were  identified  that,  if  corrected,  could  offer  significant  reductions  in  workload  and  improve  pilot  performance.  These  issues 
should  be  considered  for  future  modifications  of  the  CAAS  design,  and  future  HFEs  should  be  conducted  to  evaluate  the  effectiveness  of  any 
design  changes. 
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1.  Introduction 


1.1  Background  and  Purpose 

The  utility  helicopter  (UH)-60  Black  Hawk  is  a  twin-turbine  engine,  single  rotor,  semimonocoque 
fuselage,  rotary  wing  helicopter  capable  of  transporting  cargo,  1 1  combat  troops,  and  weapons 
during  day  and  night,  instrument  meteorological  conditions  (IMC),  visual  meteorological 
conditions  (VMC),  and  degraded  visual  enviromnent  conditions  (see  figure  1).  The  main  and  tail 
rotor  systems  consist  of  four  blades  each,  with  the  capability  to  manually  fold  the  main  rotor 
blades,  scissor  the  tail  rotor  paddles,  and  fold  the  tail  pylon  assembly  for  deployment,  transport,  or 
storage.  A  movable,  horizontal  folding  stabilator  assembly  is  situated  on  the  lower  portion  of  the 
tail  rotor  pylon  to  provide  enhanced  flight  characteristics. 


Figure  1.  UH-60M  Black  Hawk  helicopter. 

The  UH-60  Black  Hawk  helicopter  provides  air  assault,  general  support,  and  medical  evacuation 
(MEDEVAC)  capabilities  for  the  U.S.  Army.  The  UH-60  also  supports  the  Army  Airborne 
Command  and  Control  System  and  special  operations.  The  UH-60A  and  UH-60L  model  Black 
Hawk  helicopters  were  first  fielded  in  the  1970s  and  are  approaching  the  end  of  their  useful 
service  life.  Increasing  operations  and  support  costs  and  decreasing  operational  readiness  are 
consequences  of  the  aging  fleet.  The  UH-60M  program,  formerly  a  recapitalization  program  of 
existing  airframes,  is  now  a  new  production  program  designed  to  improve  the  life  of  the  current 
system,  reduce  operations  and  support  costs,  and  increase  operational  readiness.  Additionally, 
the  UH-60M  will  meet  future  digitization  and  situational  awareness  (SA)  requirements,  increase 
the  lift  and  range  capabilities  of  the  current  aircraft,  and  provide  an  improved  platform  for  the 
HH-60M  MEDEVAC  helicopter. 

General  Shoomaker,  U.S.  Army  Chief  of  Staff,  and  General  Cody,  Vice  Chief  of  Staff,  have 
challenged  the  Army  aviation  community  to  integrate  the  Common  Avionics  Architecture 
System  (CAAS),  developed  by  the  160th  Special  Operations  Aviation  Regiment  in  2003,  into 
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future  improvements  of  the  UH-60M  Black  Hawk.  The  CAAS  configuration  includes  four 
portrait-style  multi-function  displays  (MFDs),  two  control  display  units  (CDUs),  two  data 
concentrators,  and  two  general  purpose  processing  units  (see  figure  2). 


Figure  2.  Conceptual  CAAS  configuration. 


In  accordance  with  the  direction  to  integrate  CAAS,  the  UH-60M  PO  conducted  a  limited  early 
user  evaluation  (LEUE)  to  evaluate  the  integration  of  the  CAAS  in  the  UH-60M  crew  station. 
The  primary  objectives  of  this  evaluation  were  to 

•  Provide  a  system  engineering  early  risk  reduction  evaluation  to  assess  the  four  MFD 
versions  of  CAAS  in  the  UH-60M  Black  Hawk. 

•  Obtain  feedback  from  pilots  for  initial  assessments  of  the  CAAS. 

•  Provide  insight  into  emerging  capabilities  of  the  UH-60M  CAAS  cockpit. 

At  the  request  of  the  UH-60M  PMO,  the  U.S.  Army  Research  Laboratory’s  (ARL’s)  Human 
Research  and  Engineering  Directorate  conducted  a  human  factors  evaluation  (HFE)  of  the 
UH-60M  CAAS  crew  station  during  the  LEUE.  The  HFE  focused  on  workload,  SA,  and  pilot- 
vehicle  interface  (PVI).  Additional  data  collection  included  eye  tracker  data,  simulator  sickness 
data,  and  tactical  steering  committee  (TSC)  ratings. 

The  purpose  of  this  report  is  to  summarize  the  human  factors  data  collected  during  the  LEUE  by 
ARL. 

1.2  Assessment  of  Crew  Workload 

A  common  definition  of  pilot  workload  is  “the  integrated  mental  and  physical  effort  required  to 
satisfy  the  perceived  demands  of  a  specified  flight  task”  (Roscoe,  1985).  It  is  important  to  assess 
pilot  workload  because  mission  accomplishment  is  related  to  the  mental  and  physical  ability  of 
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the  crew  to  effectively  perform  their  flight  and  mission  tasks.  If  one  or  both  pilots  experience 
excessively  high  workload  while  performing  flight  and  mission  tasks,  the  tasks  may  be 
performed  ineffectively  or  abandoned.  In  order  to  assess  whether  the  pilots  are  task  overloaded 
during  the  mission  profiles,  the  level  of  workload  for  each  pilot  must  be  evaluated. 

1.2.1  Bedford  Workload  Rating  Scale 

The  pilots  completed  the  Bedford  Workload  Rating  Scale  (BWRS)  (appendix  A)  immediately 
after  each  mission  to  estimate  the  level  of  workload  that  they  experienced  during  missions.  The 
pilots  used  the  BWRS  to  rate  the  workload  needed  to  accomplish  28  UH-60M  aircrew  training 
manual  (ATM)  tasks  (appendix  A).  The  ATM  tasks  were  selected  by  personnel  from  ARL,  the 
UH-60M  PO,  and  the  U.S.  Army  Training  and  Doctrine  Command  System  Manager  (TSM)  for 
utility  aircraft  because  they  were  estimated  to  have  the  most  impact  on  pilot  workload  during  the 
planned  missions. 

The  BWRS  has  been  used  extensively  by  the  military,  civil,  and  commercial  aviation 
communities  for  pilot  workload  estimation  (Roscoe  &  Ellis,  1990).  It  requires  pilots  to  rate  the 
level  of  workload  associated  with  a  task,  based  on  the  amount  of  spare  capacity  they  feel  they 
have  to  perfonn  additional  tasks.  Spare  workload  capacity  is  an  important  commodity  for  pilots 
because  they  are  often  required  to  perform  several  tasks  concurrently.  For  example,  pilots  often 
perform  navigation  tasks,  communicate  via  multiple  radios,  monitor  aircraft  systems,  and  assist 
the  pilot  with  the  controls  to  perform  flight  tasks  (e.g.,  maintain  air  space  surveillance)  within  the 
same  time  interval.  Mission  perfonnance  is  reduced  if  pilots  are  task  saturated  and  have  little  or 
no  spare  capacity  to  perform  other  tasks.  Integration  of  the  UH-60M  CAAS  crew  station  should 
help  ensure  that  pilots  can  maintain  adequate  spare  workload  capacity  while  perfonning  flight 
and  mission  tasks. 

1.3  Assessment  of  Crew  Situational  Awareness  (SA) 

SA  can  be  defined  as  the  pilot’s  mental  model  of  the  current  state  of  the  flight  and  mission 
environment.  A  more  formal  definition  is  “the  perception  of  the  elements  in  the  environment 
within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of 
their  status  in  the  near  future”  (Endsley,  1988).  It  was  important  to  assess  SA  during  the  LEUE 
because  it  had  a  direct  impact  on  pilot  and  system  performance.  Good  SA  should  increase  the 
probability  of  good  decision  making  and  performance  by  air  crews  when  they  perform  flight  and 
mission  tasks  in  the  UH-60M  CAAS. 

1.3.1  Situation  Awareness  Rating  Technique  (SART) 

The  SART  (appendix  B)  is  a  multi-dimensional  rating  scale  for  operators  to  report  their 
perceived  SA.  The  SART  was  developed  as  an  evaluation  tool  for  the  design  of  air  crew  systems 
(Taylor,  1989)  and  examines  three  components  of  SA:  understanding,  supply,  and  demand. 
Taylor  proposed  that  SA  depends  on  the  pilot’s  understanding  (U)  (e.g.,  quality  of  information 
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he  receives)  and  the  difference  between  the  demand  (D)  on  the  pilot’s  resources  (e.g.,  complexity 
of  mission)  and  the  pilot’s  supply  (S)  (e.g.,  ability  to  concentrate).  When  D  exceeds  S,  there  is  a 
negative  effect  on  U  and  an  overall  reduction  of  SA.  The  formula  SA  =  U  -  (D  -  S)  is  used  to 
derive  the  overall  SART  score.  The  SART  is  one  of  the  most  thoroughly  tested  rating  scales  for 
estimating  SA  (Endsley,  2000). 

1.4  Assessment  of  Pilot-Vehicle  Interface  (PVI) 

The  crew  station  PVI  directly  impacts  crew  workload  and  SA  during  a  mission.  A  crew  station 
that  is  designed  to  augment  the  cognitive  and  physical  abilities  of  crews  will  minimize  workload, 
enhance  SA,  and  contribute  to  successful  mission  performance.  The  pilots  completed  a  PVI 
questionnaire  after  each  mission  (appendix  C)  to  identify  any  problems  with  the  usability  of  the 
controls,  displays,  or  subsystems. 

1.5  Assessment  of  Simulator  Sickness 

Simulator  sickness  has  been  defined  as  a  condition  where  pilots  suffer  physiological  discomfort  in 
the  simulator  but  not  while  flying  the  actual  aircraft  (Kennedy,  Lilienthal,  Berbaum,  Balzley,  & 
McCauley,  1989).  It  is  generally  believed  that  simulator  sickness  is  caused  by  a  mismatch  either 
between  the  visual  and  vestibular  sources  of  infonnation  about  self-motion,  or  between  the  sensory 
information  (e.g.,  acceleration  cues)  presented  by  the  simulator  and  the  sensory  infonnation 
presented  by  the  primary  aircraft  that  the  pilot  operates.  When  the  sensory  information  presented 
by  the  simulator  does  not  match  the  aircraft,  the  pilot’s  nervous  system  reacts  adversely  to  the 
sensory  mismatch  and  the  pilot  begins  to  experience  discomfort.  Characteristics  of  simulator 
sickness  include  nausea,  dizziness,  drowsiness,  and  several  other  symptoms  (Kennedy  et  al., 

1989).  It  is  important  to  assess  simulator  sickness  because  the  discomfort  felt  by  pilots  can  be 
distracting.  Pilot  distraction  is  one  of  the  operational  consequences  of  simulator  sickness  listed  by 
Crowley  (1987).  If  pilots  are  distracted  by  the  discomfort  they  feel  during  missions,  their 
performance  is  likely  to  suffer.  Additionally,  the  discomfort  could  influence  the  perceived  levels 
of  workload  and  SA  that  the  pilots  experienced  during  a  mission. 

1.5.1  Simulator  Sickness  Questionnaire  (SSQ) 

The  SSQ  (appendix  D)  was  administered  to  the  pilots  to  estimate  the  severity  of  physiological 
discomfort  that  they  experienced  during  missions  and  help  assess  whether  they  were  being 
distracted  by  the  discomfort.  The  SSQ  (Kennedy,  Lane,  Berbaum,  &  Lilienthal,  1993)  is  a 
checklist  of  16  symptoms.  These  symptoms  are  categorized  into  three  subscales:  oculomotor 
(e.g.,  eyestrain,  difficulty  focusing,  blurred  vision),  disorientation  (e.g.,  dizziness,  vertigo),  and 
nausea  (e.g.,  nausea,  increased  salivation,  burping).  The  three  subscales  are  combined  to 
produce  a  total  severity  score.  The  total  severity  score  is  an  indicator  of  the  overall  discomfort 
that  the  pilots  experienced  during  the  mission. 


4 


1.6  Tactical  Steering  Committee  (TSC) 

A  TSC  observed  each  mission  and  rated  crew  workload,  crew  SA,  crew  coordination,  and 
mission  success  (appendix  E).  The  TSC  provided  an  independent  assessment  of  the  workload 
and  SA  levels  experienced  by  the  crews.  They  also  helped  identify  whether  problems  with  crew 
workload  or  crew  SA  contributed  to  lack  of  mission  success. 

For  this  limited  evaluation,  only  one  subject  matter  expert  (SME)  was  available  to  serve  on  the 
TSC.  This  person  was  the  UH-60  user  representative  from  TSM-Lift  at  Fort  Rucker,  Alabama. 
He  has  substantial  experience  with  the  utility  helicopter  and  Army  aviation  missions  and  was 
very  knowledgeable  of  the  UH-60M  CAAS  crew  station  design.  He  observed  each  mission  from 
the  battle  master  station  in  the  Aviation  and  Missile  Research  and  Development  Center’s 
(AMRDEC)  Advanced  Prototyping,  Engineering,  and  Experimentation  (APEX)  Laboratory, 
Redstone  Arsenal,  Alabama,  where  he  could  observe  crew  station  displays  and  the  out-the- 
window  (OTW)  view  provided  to  the  crew.  He  also  listened  to  all  audio  communications 
between  crew  members  and  outside  sources  during  the  missions.  A  map  provided  real-time 
status  of  the  location  of  the  aircraft  on  the  terrain  database. 

1.7  AMRDEC  APEX  Laboratory 

The  UH-60M  product  manager  directed  the  use  of  AMRDEC’ s  APEX  Laboratory,  which 
provided  the  appropriate  virtual  prototyping  capabilities  required  to  perform  the  evaluation.  The 
mission  of  the  APEX  Laboratory  is  to  provide  modeling  and  simulation  support  of  weapon 
systems  early  in  the  acquisition  process.  This  is  accomplished  through  several  means,  including 
human-in-the-loop  simulators,  distributed  simulation  experimentation,  and  constructive 
simulation  development. 

The  laboratory  infrastructure  is  designed  to  support  experimentation  through  a  wide  range  of 
technologies.  The  laboratory  includes  a  battle  master  or  exercise  control  station  that  has  access 
to  each  simulation  “playing”  on  the  network  by  means  of  a  modular  semi-automated  forces 
tenninal,  data  collection  devices,  headset  communications,  and  video  monitoring  equipment.  All 
exercises  are  conducted  from  the  battle  master  station  to  ensure  that  all  players  are  engaged  in 
the  exercise  and  that  all  data  collection  devices  are  active.  The  battle  master  station  provided  the 
exercise  controller  and  tactical  steering  committee  with  all  the  information  needed  to  coordinate 
the  scenario-driven  events  and  operate  the  data  collection  devices  required  for  the  LEUE.  The 
APEX  facility  has  a  synthetic  environment  development  team  that  was  able  to  develop  custom, 
correlated  terrain  databases  that  were  designed  to  specifically  enhance  the  realism  of  the 
immersive  environment  and  support  the  operational  scenarios  for  the  event. 

1.7.1  Battlefield  Highly  Immersive  Virtual  Environment  (BHIVE) 

A  significant  component  of  the  APEX  Laboratory  is  BHIVE,  which  was  developed  in  support  of 
weapon  system  evaluation  as  a  high-level  architecture  and  distributed  interactive  simulation 
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compliant,  human-in-the-loop,  virtual  environment.  It  was  designed  with  a  “roll-in/roll-out1” 
capability  to  allow  several  types  of  devices  to  be  integrated  into  the  environment  through  a 
standard  interface.  This  capability  provides  the  flexibility  to  immerse  multiple  types  of  crew 
stations  in  a  realistic  and  re-usable  synthetic  environment. 


BHIVE  is  an  enclosed  environment  that  consists  of  a  projection  system,  three-dimensional 
surround  sound  audio,  and  a  plug-and-play  interface  for  the  integration  of  various  engineering 
analysis  devices  including  the  UH-60M  and  AH-64A  reconfigurable  crew  station,  AH-1W  and 
Z  Cobra  crew  stations,  and  a  tube-launched  optically  tracked  wire-guided  missile  simulator 
mounted  on  a  high  mobility  multipurpose-wheeled  vehicle  chassis  (see  figure  3).  The  projection 
system  consists  of  a  fixed  base,  bi-directional  curved  screen  with  three  soft-edge  blended 
projectors  and  an  image  generation  system.  The  screen  provides  a  field  of  view  of  40  degrees 
vertical  (111.61  inches)  and  150  degrees  horizontal  (229  inches).  The  distance  from  the  screen  to 
the  crew  station  is  approximately  152  inches.  BHIVE  also  includes  a  controller  station,  a  video 
switching  rack,  and  reconfigurable  video  cameras.  BHIVE  allowed  the  pilots  and  HFE  experts  to 
experiment  with  crew  station  layout  designs  and  to  perform  initial  SA  and  workload  assessments. 


Figure  3.  BHIVE  configuration. 


'  This  means  that  the  crew  station  rolls  out  of  the  “simulator”  and  a  person  can  quickly  replace  it  with  a  different 
crew  station. 
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1.7.2  The  Reconfigurable  UH-60M  Crew  Station 


The  reconfigurable  UH-60M  crew  station  was  designed  to  provide  maximum  utility  and  usability 
throughout  the  engineering  and  design  process.  The  crew  station  used  flat  panel  liquid  crystal 
displays  and  touch  screen  technology  to  replicate  human-machine  hardware  interfaces  for  the 
four  MFDs,  two  CDUs,  and  other  control  surfaces  within  the  crew  station.  This  configuration 
allowed  the  APEX  engineers  to  create  rapid  software  prototypes  of  the  actual  crew  station 
components,  based  on  the  current  design  of  the  CAAS  (see  figure  4). 


Figure  4.  CAAS  flight  displays  depicted  on  MFD. 
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2.  Method 


2.1  Participants 

Participants  consisted  of  six  pilots  from  Fort  Rucker  and  Redstone  Arsenal.  The  two  pilots  from 
Redstone  Arsenal  were  commissioned  officers  who  were  rated  in  the  UH-60  Black  Hawk  but 
were  not  currently  serving  on  flight  duty.  Of  the  pilots  from  Fort  Rucker,  two  were  senior 
warrant  officers  who  worked  as  experimental  test  pilots  at  Cairns  Anny  Airfield,  one  was  a 
warrant  officer  who  is  assigned  to  the  1-2 12th  Aviation  Regiment,  and  one  was  a  civilian  flight 
instructor  for  the  maintenance  test  pilot  course.  They  represented  a  broad  range  of  experience 
with  total  flight  hours  that  ranged  from  560  to  13,000.  The  relevant  demographic  characteristics 
of  the  pilots  are  listed  in  table  1 . 


Table  1.  Pilot  demographics  (N  =  6). 


Summary  of  demographic 
characteristics 

Age 

(yrs) 

Flight  hours  in 
UH-60A/L  Black  Hawk 

Total  flight  hours 
in  Army  aircraft 

Mean 

38.8 

1768 

3920 

Median 

38.5 

1475 

2230 

Range 

32  to  49 

360  to  3400 

560  to  13,000 

2.2  Data  Collection 

The  BWRS,  SART,  PVI,  SSQ,  and  TSC  questionnaires  (appendices  A  through  E)  were  developed 
in  accordance  with  published  guidelines  for  proper  fonnat  and  content  (O’Brien  &  Charlton, 
1996).  A  pre-test  was  conducted  to  refine  the  questionnaires  and  to  ensure  that  they  could  be 
easily  understood  and  completed  by  pilots  and  TSC  members. 

The  pilots  completed  the  PVI,  workload,  and  SA  questionnaires  immediately  after  each  mission. 
The  pilots  completed  the  SSQ  before  and  after  each  mission.  The  TSC  member  completed  the 
TSC  questionnaire  after  each  mission.  Additional  data  were  obtained  from  the  pilots  and  the  TSC 
member  during  post-mission  discussions  and  the  final  after-action  review  (AAR).  Questionnaire 
results  were  clarified  with  infonnation  obtained  during  post-mission  discussions  and  the  daily 
AARs. 

2.2.1  Eye  Tracker  System 

Although  the  data  from  the  questionnaires  were  systematically  gathered  by  widely  accepted  HFE 
methods,  they  were  still  subjective  in  nature.  Complementary  objective  data  were  collected 
through  a  head  and  eye  tracking  system  from  Applied  Science  Laboratories  (ASL).  Their  system 
was  used  because  it  was  capable  of  integrating  a  laser  head  tracker  to  allow  unrestricted  head 
movement  during  data  collection,  and  it  was  compatible  with  the  head  gear  unit-56  flight  helmet. 
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The  ASL  EyeHead2  Package  integrated  a  Model  501  eye  tracker  and  an  Ascension  Laserbird3 
head  tracker.  This  technology  allowed  us  to  collect  digital  data  that  specified  point  of  gaze  with 
respect  to  stationary  objects  within  the  crew  station.  The  ASL  software  allowed  data  collectors 
to  continuously  monitor  the  eye  position  of  the  pilots  by  crosshairs  superimposed  over  live 
imagery  (see  figure  5).  The  software  also  included  a  built-in  analysis  tool  that  allowed  data  to  be 
viewed  in  tabular  or  graphical  format. 


Figure  5.  Eye  tracker  scene  camera  monitors  and  control  panel  interface. 


2.3  Data  Analysis 

Pilot  responses  to  the  BWRS,  SART,  SSQ,  and  PVI  questionnaires  were  analyzed  with  means 
and  percentages.  Their  responses  to  the  BWRS,  SART,  and  SSQ  were  further  analyzed  with  the 
Wilcoxon  Signed  Ranks  Test  (WSRT)  to  compare  pilot  ratings  between  seating  position  (left 
versus  right).  The  WSRT  was  used  to  calculate  probability  values  for  data  comparisons. 

We  summarized  the  eye  tracker  data  by  calculating  the  total  percentage  of  fixations  that  were 
focused  on  different  areas  of  interest  (AOI).  Four  AOIs  were  created  for  each  pilot:  left  MFD, 
right  MFD,  CDU,  and  OTW  (see  figure  6).  A  final  category,  called  “Other,”  captured  eye 
fixations  not  focused  on  a  specific  AOI.  Most  of  the  fixations  captured  in  the  “Other”  category 


9  . 

EyeHead  is  a  trademark  of  Applied  Science  Laboratories. 
Laserbird  is  a  trademark  of  Ascension  Technology  Corporation. 
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resulted  from  the  pilot  looking  at  his  kneeboard  or  looking  across  the  cockpit  at  the  other  pilot’s 
displays. 


Figure  6.  Eye  tracker  areas  of  interest. 


2.4  Limitations  of  Assessment 

The  LEUE  had  several  limitations  that  restricted  the  human  factors  experts  from  performing  a 
full  assessment  of  all  aspects  of  the  CAAS  crew  station.  Only  the  right  crew  station  seat  was 
equipped  with  functional  flight  controls.  Therefore,  all  flying  was  done  from  the  right  seat.  In 
real-life  missions,  pilots  would  likely  alternate  flying  responsibilities  periodically.  In  addition, 
the  flight  controls  (i.e.,  the  collective  and  cyclic)  were  not  representative  of  the  actual  CAAS 
hardware.  This  limited  the  ability  to  evaluate  the  usability  characteristics  of  the  flight  controls 
during  the  missions. 

All  four  MFDs  and  the  two  CDUs  were  simulated  with  touch  screens  controlled  by  computers 
outside  the  BHIVE.  A  comprehensive  usability  assessment  could  not  be  completed  since  these 
components  were  not  production-representative  hardware;  however,  the  configuration  provided 
an  adequate  interface  to  evaluate  SA,  workload,  and  the  design  of  displays. 

Both  the  multi-function  control  unit  (MFCU)  and  multi-function  knob  (MFK)  were  limited  in 
functionality.  The  limitations  of  the  MFK  were  not  significant;  however,  the  limited 
functionality  of  the  MFCU  was  significant  and  resulted  in  a  very  limited  amount  of  data  being 
collected  regarding  usability. 
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The  tactical  data  link  (TDL)  messaging  also  posed  some  limitations.  Only  three  types  of 
messages  were  available  for  use  during  the  LEUE.  These  messages  consisted  of  position  reports, 
free-text  messages,  and  observation  reports.  Also,  the  TDL  system  was  limited  to  transmitting 
capabilities  only  and  did  not  allow  pilots  to  receive  incoming  messages.  This  limitation  resulted 
in  lower  than  normal  workload  associated  with  managing  communications  during  operational 
flights. 

Several  other  limitations  existed  with  the  software  representation  of  the  CAAS  cockpit.  Some  of 
the  software  limitations  that  impacted  this  evaluation  were  the  lack  of  a  fully  functional  flight 
path  stabilization  system.  The  CDUs  did  not  share  data  as  intended,  the  mission  and  flight 
planning  pages  were  not  fully  developed,  and  the  CDU  return  button  on  the  flight  director 
overlay  was  inoperable.  These  limitations  are  not  uncommon  when  one  is  replicating  a  complex 
aviation  system  in  a  prototype  simulator;  however,  they  are  worthy  of  noting  because  we  suspect 
they  had  some  impact  on  the  results  of  this  evaluation. 

The  order  of  the  scenarios  combined  with  the  method  of  crew  rotation  presented  a  unique 
limitation  for  the  eye  tracker  data.  Since  the  BHIVE  could  only  be  flown  from  the  right  seat,  the 
crews  rotated  seats  for  each  mission  scenario.  Since  there  were  three  missions,  the  same  three 
pilots  were  in  the  right  seat  for  scenario  1  and  scenario  3.  These  two  scenarios  were  the  two 
visual  flight  rules  (VFR)  flights  for  which  eye  tracker  data  were  collected.  Because  of  this 
conflict,  eye  tracker  data  for  the  flying  pilot  in  the  right  seat  were  limited  to  a  sample  size  of 
three  instead  of  six. 

The  LEUE  was  intended  to  be  a  limited  evaluation  of  the  CAAS  design.  As  such,  the  evaluation 
used  a  limited  number  of  participants  and  only  executed  a  small  number  of  short  operational 
scenarios. 

2.5  Test  Schedule  and  Description  of  Mission  Scenarios 

The  entire  evaluation  occurred  over  a  two-week  time  period  in  October  and  November  2004  and 
consisted  of  two  phases.  The  first  phase  of  the  evaluation  was  pilot  training.  This  was  conducted 
by  members  of  the  AMRDEC  SSDD  and  lasted  one  week.  During  this  training,  pilots  received  a 
complete  overview  of  the  CAAS  and  were  told  of  any  simulation  limitations.  The  training  was 
conducted  with  classroom  instruction,  desktop  trainers,  and  the  BHIVE.  The  second  phase  of  the 
evaluation  was  the  week  of  testing.  Day  1  consisted  of  a  welcome  briefing  given  by  the  UH-60M 
PO  and  a  briefing  for  the  first  mission.  The  second  day  was  the  first  of  three  days  dedicated  to 
flying  mission  scenarios  and  data  collection.  The  last  day  of  the  evaluation  consisted  of  a  final 
AAR  and  briefing. 

The  trials  consisted  of  three  mission  scenarios  developed  by  the  Directorate  of  Combat 
Developments  UH-60  user  representative  from  Fort  Rucker.  The  scenarios  were  developed  in 
accordance  with  UH-60  tactics,  techniques,  and  procedures.  The  mission  scenarios  were  VMC 
flight,  IMC  flight,  and  tactical  flight.  Figures  7  through  9  show  the  flight  plans  for  each  mission 
scenario. 
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Figure  7.  Scenario  1:  visual  flight  rules  flight  plan. 


Figure  8.  Scenario  2:  instrument  flight  rules  flight  plan. 
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Figure  9.  Scenario  3:  tactical  flight  plan. 


Three  sets  of  air  crews  (six  pilots)  performed  each  mission  scenario.  The  schedule  consisted  of 
all  three  crews  flying  the  same  mission  in  a  single  day.  This  schedule  resulted  in  all  three 
missions  being  completed  in  three  days  for  a  total  of  nine  flights.  The  pilots  received  crew 
assignments  and  a  mission  briefing  at  the  end  of  each  day  for  the  next  day’s  mission,  allowing 
them  to  plan  and  rehearse  at  their  discretion  in  the  evening  or  the  next  morning. 

Each  mission  concluded  with  the  crews  completing  the  human  factors  surveys  in  a  nearby  office. 
After  all  three  crews  completed  the  mission  and  surveys,  they  participated  in  a  mission 
debriefing  and  AAR.  At  the  end  of  the  week,  the  pilots  participated  in  a  final,  comprehensive 
AAR. 


3.  Results 


3.1  Crew  Workload 

3.1.1  Mean  Workload  Ratings  for  ATM  Tasks 

The  mean  overall  workload  rating  for  all  ATM  tasks  was  3.10.  The  workload  ratings  were  also 
looked  at  individually  for  each  seat  since  flight  controls  were  only  available  for  the  right  seat  of 
the  BHIVE.  The  mean  workload  rating  was  3.33  for  the  left  seat  and  2.98  for  the  right  seat  (see 
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figure  10).  These  ratings  indicate  that  the  pilots  typically  had  “enough  workload  capacity  for  all 
desirable  additional  tasks”  while  perfonning  crew  duties  from  either  seating  position.  The 
difference  in  workload  ratings  between  the  left  and  right  seats  was  statistically  significant 
(WSRT,  z  =  -2.201  ,p  =  0.028);  however,  the  practical  difference  is  not  significant  because  both 
ratings  are  clustered  around  the  “3”  on  the  BWRS. 

Two  tasks  received  peak  workload  ratings  that  indicated  that  workload  was  not  tolerable  for  the 
task.  The  mean  and  peak  workload  ratings  for  responding  to  inadvertent  instrument  meteoro¬ 
logical  conditions  (IIMC)  were  4.33  and  8.00.  The  mean  and  peak  workload  ratings  for  operating 
the  CDU  were  4.19  and  7.00.  The  mean  ratings  indicate  that  the  workload  required  for  these 
tasks  resulted  in  insufficient  spare  workload  capacity  for  easy  attention  to  additional  tasks; 
however,  the  peak  ratings  indicate  that  there  were  moments  of  intolerable  workload  associated 
with  these  tasks.  Pilots  indicated  that  they  experienced  high  workload  for  these  tasks  because  too 
many  steps  were  required  to  set  critical  settings  in  the  CDU.  Peak  workload  is  noteworthy 
because  even  short  periods  of  intolerable  workload  can  overwhelm  the  pilots  and  create  the 
potential  for  pilot  error.  Peak  workload  ratings  higher  than  7.00  should  be  identified  and  resolved 
through  crew  station  design.  Appendix  F  includes  a  table  of  mean  workload  ratings  for  all  tasks. 
A  set  of  pilot  comments  regarding  workload  is  included  in  appendix  G. 


Workload  Ratings  on  ATM  Tasks  for  Left  and 
Right  Seats 


Figure  10.  Workload  ratings  on  ATM  tasks  for  left  and  right  seats. 


3.1.2  TSC  Ratings  for  Workload  and  Crew  Coordination 

The  TSC  provided  an  overall  mean  workload  rating  of  2.89  for  pilots  sitting  in  either  crew 
position — a  lower  rating  than  the  pilots  gave  themselves.  An  overall  mean  rating  of  2.89  also 
indicates  that  the  pilots  typically  had  “enough  workload  capacity  for  all  desirable  additional 
tasks”.  The  TSC  also  gave  individual  ratings  for  pilots  in  each  seat.  The  workload  ratings  for 
the  left  and  right  seats  were  3.00  and  2.78,  respectively.  TSC  comments  indicated  that  the  left 
seat  was  assigned  higher  workload  ratings  because  too  many  button  pushes  were  required  to 
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perform  basic  co-pilot  functions  (e.g.,  systems  check,  before  take-off  check,  initiate  and  “close¬ 
out”  fuel  consumption  check,  and  monitor  fuel  throughout  the  mission). 

The  TSC  also  rated  crew  coordination  for  each  mission  using  a  5-point  rating  scale.  The  mean 
crew  coordination  rating  for  all  missions  was  2.44  (see  figure  1 1).  The  TSC  did  not  provide 
comments  regarding  why  he  rated  crew  coordination  the  way  he  did. 


Mean  Crew 
Coordination 
Rating  (2.44) 

r 

1  2  3  4  5 

Excellent  Good  Average  Needs  Improvement  Unacceptable 

Figure  11.  Overall  TSC  crew  coordination  ratings. 


3.2  Crew  Situational  Awareness 

3.2.1  Situational  Awareness  Ratings  by  the  Subjects 

The  overall  SART  score  provided  by  the  pilots  was  25.84  for  the  UH-60M  CAAS.  This  score 
indicates  that  the  pilots  felt  they  had  moderate  levels  of  overall  SA  during  the  missions.  The  SA 
ratings  for  the  left  and  right  seats  were  26.42  and  25.25,  respectively  (see  figure  12).  The 
difference  between  SA  scores  for  the  left  and  right  seats  were  not  statistically  significant 
(WSRT,  z  =  -0.108,/?  =  0.914). 


Figure  12.  Overall  SART  scores  for  left  and  right  seats. 


The  SA  ratings  were  further  analyzed  to  compare  the  difference  in  scores  between  the  VFR  and 
instrument  flight  rules  (IFR)  missions.  This  comparison  is  particularly  important  because  the 
flying  pilot  is  completely  dependent  on  the  information  provided  by  the  aircraft  displays  during 
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IFR  flight.  The  mean  SART  scores  for  the  left  seat  pilots  for  VFR  and  IFR  missions  were  22.83 
and  30.00,  respectively.  The  mean  SART  scores  for  the  right  seat  pilots  for  VFR  and  IFR  missions 
were  26.17  and  24.33,  respectively.  Neither  of  these  results  was  statistically  significant;  however, 
the  results  indicate  that  the  left  seat  pilots  experienced  higher  SA  during  IFR  flights  while  the  right 
seat  pilot  experienced  lower  SA  during  IFR  flights.  Figures  13  and  14  show  the  comparison  of 
subscale  ratings  by  pilot  for  VFR  and  IFR  flights.  The  subscale  ratings  show  that  the  left  seat  pilot 
experienced  an  increase  in  supply  during  IFR  missions  while  the  right  seat  pilot  experienced  a 
decrease  in  supply  during  IFR  missions.  Appendix  H  lists  all  pilot  comments  regarding  SA. 


Figure  13.  Comparison  of  VFR  with  IFR  SA  for  left  seat. 
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Figure  14.  Comparison  of  VFR  with  IFR  SA  for  right  seat. 


3.2.2  TSC  Situational  Awareness  Ratings 

The  TSC  provided  an  independent  assessment  of  SA  based  on  the  scale  shown  in  table  2.  The 
mean  TSC  SA  rating  was  1.78.  This  indicates  that  the  TSC  perceived  that  the  crews  typically  had 


adequate  levels  of  SA  with  some  periods  of  minor  variation  between  perception  and  reality.  A 
complete  set  of  TSC  comments  regarding  SA  is  presented  in  appendix  I. 


Table  2.  TSC  situational  awareness  ratings. 


1 

Crew  was  consistently  aware  of  all  entities  on  the  battlefield. 1 

2 

Crew  was  aware  of  the  battlefield  with  minor  or  insignificant  variation  between  perception  and  reality. 

3 

Crew  was  aware  of  the  battlefield.  Variation  between  reality  and  perception  did  not  significantly  impact 
mission  success. 

4 

SA  needs  improvement.  Lack  of  SA  had  some  negative  effect  on  the  success  of  the  mission. 

5 

Lack  of  SA  caused  mission  failure. 

‘Mean  rating  1.78  (SD  =  0.67) 

3.3  PVI 


The  pilots  completed  a  comprehensive  PVI  survey  after  each  mission.  This  survey  allowed  the 
pilots  to  assign  ratings  for  each  question  and  provide  comments  on  why  they  rated  the  question  a 
certain  way.  This  section  of  the  report  highlights  some  of  the  most  common  issues  that  were 
identified  by  the  pilots,  based  on  the  results  of  the  survey  or  the  post-mission  AAR.  A  complete 
set  of  PVI  results  is  included  in  appendix  J. 

3.3.1  Multi-Function  Displays 

Pilots  were  generally  comfortable  with  the  symbology  used  on  the  MFD  pages  in  the  CAAS 
cockpit;  however,  they  reported  several  problems  with  the  vertical  situation  display  (VSD), 
engine  instrument  caution  advisory  system  (EICAS),  and  digital  map. 

Sixty-one  percent  of  the  pilot  responses  indicated  that  they  experienced  problems  using  the  VSD 
during  the  missions.  One  design  feature  that  pilots  identified  as  a  major  safety  concern  was  the 
angle-of-bank  indicator.  This  indicator  is  situated  on  the  top  of  the  VSD  and  displays  the  angle 
of  aircraft  bank  when  the  pilot  enters  a  turn  (see  figure  15).  The  design  of  the  angle-of-bank 
indicator  in  the  CAAS  cockpit  works  differently  from  angle-of-bank  indicators  in  all  other  Army 
aircraft.  Even  after  being  trained  to  use  the  angle-of-bank  indicator  and  knowing  that  it  worked 
differently,  pilots  often  increased  their  angle  of  bank  when  they  intended  to  roll  out  of  a  turn. 

Another  issue  the  pilots  identified  with  the  VSD  was  that  there  was  no  fuel  indicator  on  this 
“page”4.  Fuel  is  a  critical  variable  that  needs  to  be  monitored  constantly  during  flight.  The 
current  procedure  in  the  CAAS  cockpit  to  check  the  aircraft’s  fuel  level  requires  several  button 
pushes  on  the  MFD  to  access  the  fuel  status.  The  pilots  indicated  that  this  procedure  was  too 
cumbersome  for  them  to  perfonn  a  routine  task. 

The  lack  of  a  Kollsman  window  was  also  identified  as  a  shortcoming  of  the  CAAS  design.  A 
Kollsman  window  is  a  display  used  on  other  Army  aircraft  to  display  the  four-digit  altimeter 


4The  aircraft’s  multi-function  displays  have  the  flexibility  to  display  multiple  “pages.” 
that  can  be  displayed. 


The  VSD  is  one  “page” 
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setting.  The  pilots  must  frequently  change  the  barometric  altimeter  setting  because  the 
barometric  pressure  changes  frequently.  The  C AAS  does  not  use  a  Kollsman  window.  Instead, 
this  information  is  situated  on  the  CDU.  Pilots  indicated  that  the  procedure  required  to  check 
and  modify  the  altimeter  setting  was  workload  intensive  and  they  recommended  the  use  of  a 
Kollsman  window  on  the  MFD. 


Figure  15.  VSD  angle  ofbank  indicator. 

Seventy-eight  percent  of  the  pilot  responses  indicated  that  they  experienced  problems  using  the 
EICAS  during  the  missions.  The  most  common  complaint  by  the  pilots  was  that,  given  the 
current  design,  they  could  not  monitor  all  required  instruments  and  systems  simultaneously. 
Each  pilot  has  two  MFDs  that  can  be  split  to  display  a  total  of  four  different  MFD  pages. 
Typically,  the  flying  pilot  must  monitor  the  VSD  and  horizontal  situation  indicator  (FISD)  on 
one  MFD.  On  the  other  MFD,  the  pilots  usually  displayed  the  digital  map  and  the  engine  and 
transmission  systems.  At  this  point,  all  four  sections  are  being  occupied  and  the  pilot  does  not 
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have  the  ability  to  see  fuel  without  additional  button  pushes  on  the  MFD.  The  pilots  recom¬ 
mended  adding  fuel  to  the  VSD  or  incorporating  all  aircraft  systems  on  a  single  El  CAS  page. 

Sixty-one  percent  of  the  responses  reported  that  the  pilots  experienced  problems  using  the  digital 
map.  The  light  colors  of  the  HSD,  route  lines,  cursor,  and  aircraft  symbol  are  very  difficult  to 
see  when  they  are  overlaid  on  the  light  digital  map.  Pilots  recommended  changing  the  color  of 
these  features  in  order  to  make  them  more  visible  on  the  digital  map. 

3.3.2  Multi-Function  Knob  (MFK) 

Only  22%  of  the  pilot  responses  indicated  that  they  experienced  problems  using  the  MFK; 
however,  only  1 1%  of  the  responses  reported  that  pilots  preferred  using  the  MFK  over  manually 
typing  settings  into  the  CDU.  Pilots  reported  that  the  process  of  navigating  through  the  CDU  in 
order  to  activate  the  function  of  the  MFK  was  workload  intensive.  Most  pilots  indicated  that 
after  performing  the  required  steps  on  the  CDU  to  activate  the  MFK,  they  would  prefer  to 
continue  entering  digits  in  the  CDU.  The  pilots  recommended  that  the  MFK  be  linked  to  “hot 
keys”  on  the  MFD  overlay  pages,  which  would  easily  activate  the  MFK  for  specific  functions 
with  a  single  button  push  or  that  multiple  knobs  be  used — each  with  its  own  specific  function. 

3.3.3  Central  Display  Unit  (CDU) 

The  pilots  did  not  report  many  problems  setting  the  initial  settings  in  the  CDU;  however,  they 
did  report  problems  managing  global  positioning  system/flight  plan  (GPS/FP)  and 
communication/navigation  (COM/NAV)  information  in  the  CDU  during  the  missions.  The 
pilots  indicated  that  the  CDU  was  difficult  to  use  because  many  common  functions  were  too 
deep  in  the  menu  structure.  The  pilots  gave  several  examples  of  tasks  that  were  too  deep  in  the 
CDU,  such  as  changing  the  altimeter  setting,  selecting  a  new  heading,  and  changing  air  control 
point  (ACP)  locations.  In  most  cases,  the  pilots  commented  that  the  menu  structure  for  the  CDU 
was  not  intuitive  and  it  caused  them  to  forget  where  certain  functions  were.  We  suspect, 
however,  that  a  large  percentage  of  these  problems  were  caused  by  the  inappropriate  or 
incomplete  implementation  of  the  CDU  in  the  BHIVE  and  that  if  the  pilots  were  given  a 
production-representative  CDU,  many  of  these  problems  would  be  eliminated. 

3.3.4  Other  PVI  Issues 

•  The  zoom  feature  on  the  digital  map  was  difficult  to  use. 

•  The  needles  on  the  HSD  should  be  different  colors  so  they  are  easier  to  distinguish. 

•  The  heading  select  marker  should  have  the  capability  to  be  selected  without  engaging  the 
flight  director. 
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•  The  font  size  of  the  letters  and  numbers  on  the  HSD  was  too  small  and  difficult  to  read.  In 
particular,  the  cardinal  directions  (i.e.,  north,  south,  east,  west)  should  stand  out  so  it  is  easy 
to  determine  the  aircraft’s  general  direction  at  a  glance. 

3.4  Simulator  Sickness 

Pilots  reported  very  few  simulator  sickness  symptoms  during  the  LEUE.  Most  of  the  symptoms 
involved  slight  sweating,  general  discomfort,  fatigue,  and  mild  eyestrain.  The  overall  mean  total 
severity  score  (post  mission)  for  the  pilots  was  9. 15  (see  table  3).  The  simulator  sickness  scores 
were  also  compared  for  VMC  versus  IMC  missions.  The  total  severity  scores  for  VMC  and  IMC 
missions  were  12.15  and  6.23,  respectively.  These  results  show  a  large  difference;  however,  the 
difference  was  not  statically  significant  (WSRT,  z  =  - 1 .753,  =  0.08).  Overall,  the  BHIVE 

posed  no  problems  for  simulator  sickness  and  should  continue  to  be  very  suitable  as  a  simulation 
environment  in  the  future. 


Table  3.  Simulator  sickness  questionnaire  ratings. 


Condition 

Nausea 

Subscale 

Oculomotor 

Subscale 

Disorientation 

Subseale 

Total  Severity 
Score  (Mean) 

SD 

Pre-Mission 

2.64 

3.35 

3.87 

3.73 

4.59 

Post  Mission 

6.36 

11.81 

3.09 

9.15 

7.39 

VMC 

9.54 

14.53 

4.64 

12.15 

9.37 

IMC 

4.77 

6.32 

4.64 

6.23 

10.22 

SD  =  standard  deviation 


3.4.1  Comparison  of  BHIVE  SSQ  Scores  to  Other  Helicopter  Simulators 

To  assess  whether  the  SSQ  ratings  provided  by  the  pilots  during  the  LEUE  were  similar  to  or 
different  from  ratings  obtained  in  other  helicopter  simulators,  the  mean  total  severity  scores  for  the 
BHIVE  were  compared  to  the  mean  total  severity  scores  for  several  other  helicopter  simulators 
(AH-64A,  S-3H,  CH-46E,  CH-56D,  CH-56F,  Sikorsky  reconnaissance  attack  helicopter  [RAH]-66 
engineering  development  simulator  [EDS],  RAH-66  Comanche  portable  cockpit  [CPC],  and  the 
BHIVE  from  the  early  user  demonstration  [EUD])  (see  table  4).  These  simulators  typically 
induced  low  to  moderate  levels  of  simulator  sickness  symptoms  in  pilots. 


Table  4.  Comparison  of  BHIVE  SSQ  ratings  with  other  helicopter  simulators. 


Simulator 

Nausea 

Subseale 

Oculomotor 

Subseale 

Disorientation 

Subscale 

Total  Severity  Score 
(Mean) 

AH-64A* 

25.81 

SH-3H 

14.70 

20.00 

12.40 

18.80 

RAH-66  EDS 

11.84 

14.98 

4.54 

13.25 

CH-53F 

7.50 

10.50 

7.40 

10.00 

RAH-66  CPC 

3.29 

12.94 

7.89 

9.80 

UH-60  BHIVE  (LEUE) 

6.36 

11.81 

3.09 

9.15 

UH-60M  BHIVE  (EUD) 

13.88 

6.89 

0 

8.50 

CH-53D 

7.20 

7.20 

4.00 

7.50 
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CH-46E 

5.40 

7.80 

4.50 

7.00 

*SSQ  subscale  data  not  available. 


3.5  Eye  Tracker 

The  importance  of  collecting  eye  tracker  data  was  to  determine  how  well  the  design  of  the 
aircraft  allowed  the  flying  pilot  to  remain  focused  outside  the  aircraft  during  VFR  flight.  During 
the  missions  conducted  during  the  LEUE,  there  were  several  instances  when  the  pilots 
momentarily  landed  to  pick  up  or  drop  off  troops.  Eye  tracker  data  collected  during  these 
periods  of  the  experiment  were  eliminated  from  the  analysis  of  the  eye  tracker.  Scenario  2  was 
an  IFR  flight.  During  IFR  flight,  pilots  remain  focused  inside  the  aircraft  and  flew  the  aircraft 
based  on  the  aircraft  instruments.  Eye  tracker  data  were  not  collected  during  scenario  2.  Finally, 
the  BHIVE  was  limited  to  being  flown  from  the  right  seat  only.  In  a  realistic  scenario,  pilots 
would  occasionally  transfer  flight  controls  so  they  could  safely  perform  other  duties.  During  the 
LEUE,  this  was  not  possible.  This  is  important  to  point  out  because  significant  differences  in 
eye  tracker  data  are  common  between  the  flying  and  non- flying  pilots.  In  this  case,  the  flying 
pilot  was  always  in  the  right  seat. 

Table  5  shows  the  percentage  of  time  that  the  left  and  right  seat  pilots  were  focused  on  each  area 
of  interest  defined  in  the  crew  station.  The  “other”  category,  also  defined  in  the  method  section 
of  this  report,  included  periods  of  time  when  the  pilots  were  focused  in  an  area  not  defined  by  an 
area  of  interest.  Examples  include  looking  down  at  their  kneeboard  or  glancing  across  the 
cockpit  to  see  the  other  pilot’s  displays. 


Table  5.  Summary  of  eye  tracker  results  for  left  and  right  seats. 


Out  the  Window 

Left  MFD 

Right  MFD 

CDU 

Other 

Left  Seat 

26.30% 

13.85% 

35.68% 

7.89% 

16.27% 

Right  Seat 

60.86% 

8.51% 

27.20% 

0.05% 

3.39% 

Figure  16  shows  the  eye  tracker  data  in  a  graphical  fonnat. 

The  data  indicated  that  the  right  seat  pilot  spent  an  unusually  high  percentage  of  time  focused  on 
the  right  MFD.  This  anomaly  motivated  us  to  further  analyze  the  data  to  detennine  the  cause  of 
this  problem.  We  determined  that  a  significant  difference  in  results  between  subjects  was  the 
cause  of  the  unusual  data.  Two  of  the  subjects  were  Aviation  Technical  Test  Center  (ATTC) 
experimental  test  pilots.  Table  6  shows  a  comparison  of  eye  tracker  results  for  the  ATTC  pilots 
and  non- ATTC  pilots.  The  comparison  indicates  that  the  ATTC  pilots  spent  a  much  larger 
percentage  of  time  focused  on  the  right  MFD  than  the  non- ATTC  pilots.  After  consulting  with 
the  ATTC  pilots  and  several  other  SMEs,  we  concluded  that  this  was  caused  by  the  specialized 
training  and  experience  of  experimental  test  pilots.  Experimental  test  pilots  are  highly 
experienced  pilots  who  are  trained  to  perfonn  advanced  test  flight  maneuvers  in  the  aircraft  and 
closely  monitor  the  aircraft  systems  during  these  maneuvers. 
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Figure  16.  Graphical  representation  of  the  eye  tracker  results. 


Table  6.  Comparison  of  eye  tracker  results  from  ATTC  versus  non-ATTC  pilots  (right  seat). 


Out  the  Window 

Left  MFD 

Right  MFD 

CDU 

Other 

ATTC 

52.06% 

6.34% 

38.02% 

0.00% 

3.59% 

Non-ATTC 

75.58% 

12.14% 

9.11% 

0.13% 

3.05% 

3.5.1  Comparison  of  Eye  Tracker  Data  From  Previous  UH-60  Assessments 

The  LEUE  was  the  third  of  a  series  of  evaluations  conducted  to  evaluate  the  suitability  of  the  UH- 
60M  crew  station.  Eye  tracker  data  were  also  collected  during  the  previous  two  UH-60  evalua¬ 
tions,  the  EUD2  and  the  limited  user  test  (LUT).  Table  7  shows  a  comparison  of  eye  tracker  data 
from  each  of  the  three  evaluations.  Although  a  comparison  of  the  results  of  each  evaluation  is 
useful,  one  must  be  cautious  to  remember  that  each  set  of  data  was  collected  in  different  evalua¬ 
tions  that  were  all  conducted  differently.  For  example,  the  scenarios  flown  in  each  evaluation  were 
different,  the  areas  of  interest  were  defined  differently  in  each  evaluation,  and  a  different  number 
of  participants  with  different  experience  levels  participated  in  each  evaluation. 


Table  7.  Comparison  of  eye  tracker  results  from  EUD2,  LUT,  and  LEUE. 


EUD2 

LUT 

LEUE 

Flying  Pilot 

Non-Flying 

Pilot 

Flying  Pilot 

Non-Flying 

Pilot 

Flying  Pilot 

Non-Flying 

Pilot 

Outside 

69.21% 

N/A 

85.60% 

28.21% 

60.86% 

26.30% 

Inside 

30.79% 

N/A 

14.40% 

71.79% 

39.14% 

73.70% 
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4.  Summary 


4.1  Summary  of  Crew  Workload 

•  Pilots  reported  a  mean  workload  rating  of  3. 10.  This  indicates  that  there  was  enough 
workload  capacity  for  all  desirable  tasks. 

•  Three  ATM  tasks  received  peak  workload  ratings  of  6.  Tasks  with  a  workload  rating  of  6 
indicate  that  the  level  of  effort  required  allows  little  attention  to  additional  tasks.  These 
tasks  were  maintain  air  space  surveillance,  perform  instrument  maneuvers,  and  operate  the 
MFDs. 

•  Two  tasks  received  peak  workload  ratings  of  greater  than  6.  A  workload  rating  higher  than 
6  indicates  that  workload  was  not  tolerable  for  the  task.  These  tasks  were  respond  to 
inadvertent  IMC  and  operate  CDU. 

•  The  TSC  rated  the  average  pilot  workload  at  2.89,  indicating  the  pilots  had  enough 
workload  capacity  for  all  desirable  tasks. 

4.2  Summary  of  Crew  Situational  Awareness 

•  Pilots  reported  a  mean  SA  rating  of  25.84,  with  very  little  difference  between  seat  position. 
This  indicates  that  the  pilots  felt  they  had  moderate  levels  of  SA  during  missions. 

•  The  flying  pilot  reported  slightly  higher  SA  during  the  VMC  missions,  while  the  non-flying 
pilot  reported  having  higher  SA  during  IMC  missions. 

•  SA  seemed  to  be  influenced  by  the  information  displayed  on  the  MFDs.  Pilots  reported 
having  difficulty  determining  the  status  of  the  aircraft  at  all  times  (i.e.,  monitoring  aircraft 
systems  and  fuel,  reading  the  compass  on  the  digital  map,  and  determining  the  direction  of 
aircraft  bank  using  the  bank  indicator  on  the  VSD). 

•  The  TSC  rated  crew  SA  as  1.78.  This  indicates  that  the  TSC  felt  the  crew  was  aware  of  the 
battlefield  and  their  own  ship  with  minor  or  insignificant  variation  between  perception  and 
reality. 

4.3  PVI 

•  The  pilots  did  not  feel  that  all  required  infonnation  was  displayed  on  the  VSD.  Pilots 
commented  that  fuel,  Nr  (rotor  revolutions  per  minute  [rpm]),  Ng  (gas  turbine  rpm),  and 
turbine  gas  temperature  (TGT)  should  be  located  on  the  VSD.  They  also  requested  that  the 
VSD  display  a  Kollsman  window  and  main  fuel. 
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•  The  roll  rate  indicator  on  the  VSD  indicates  the  direction  of  roll  opposite  from  the  current 
UH-60  fleet.  This  caused  disorientation  in  some  of  the  pilots  and  they  raised  this  as  a 
safety  concern. 

•  Pilots  overwhelmingly  agreed  that  all  engine  and  transmission  systems,  fuel,  and  cautions 
and  advisories  should  be  on  a  single  MFD  page. 

•  The  pilots  indicated  that  the  process  for  verifying  and  changing  the  altimeter  setting 
required  too  many  steps. 

•  Pilots  noted  that  the  contrast  between  the  digital  map  and  the  overlay  was  not  adequate, 
causing  difficulty  when  they  read  the  overlay  data. 

•  The  crews  indicated  that  the  CDU  menu  structure  was  complicated  and  non-intuitive.  They 
reported  that  using  the  CDU  required  them  to  be  “inside”  too  much. 

•  In  general,  the  crews  did  not  benefit  from  using  the  MFK.  The  process  for  selecting  the 
MFK  function  was  time  consuming  and  was  performed  on  the  CDU.  Pilots  felt  more 
comfortable  finishing  the  process  on  the  CDU  rather  than  switching  to  the  MFK. 

4.4  Simulator  Sickness 

•  Pilots  reported  mild  simulator  sickness  symptoms  after  flying  missions  in  the  BHIVE. 
However,  pilots  reported  higher  severity  scores  after  VMC  flight.  The  severity  scores  were 
6.23  for  IMC  missions  and  12.15  for  VMC  missions. 

•  Simulator  sickness  symptoms  did  not  adversely  affect  pilot  performance. 

4.5  Eye  Tracker 

•  The  flying  pilot  typically  spent  60.86%  of  the  time  fixated  OTW,  27.2%  fixated  on  the 
right  MFD,  8.5%  on  the  left  MFD,  and  only  0.05%  on  the  CDU.  The  remaining  time  was 
spent  looking  at  other  areas  that  were  not  specifically  examined  during  this  evaluation  (e.g., 
kneeboard,  center  console,  etc.). 

•  The  non-flying  pilot  spent  26.3%  fixated  OTW,  35.68%  on  the  right  MFD,  13.8%  on  the 
left  MFD,  and  7.9%  on  the  CDU.  The  remaining  time  was  spent  looking  at  other  areas  that 
were  not  specifically  examined  during  this  evaluation  (e.g.,  kneeboard,  center  console, 
etc.). 

•  A  significant  difference  was  noted  between  OTW  percentages  for  ATTC  versus  non-ATTC 
pilots.  The  average  OTW  percentage  for  ATTC  pilots  was  52.1%  and  the  average  OTW 
percentage  for  non-ATTC  pilots  was  75.6%.  This  significant  difference  contributed  to  the 
fact  that  ATTC  experimental  test  pilot  (XP’s)  spend  more  time  monitoring  the  status  of 
aircraft  systems  during  flight,  compared  to  non-ATTC  pilots. 
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5.  Recommendations 


The  following  recommendations  are  made  to  enhance  the  overall  effectiveness  and  suitability  of 
the  UH-60M  CAAS  crew  station  as  it  continues  its  development: 

•  Address  and  resolve  the  workload  and  PVI  issues  identified  during  the  LEUE. 

•  Use  the  Crew  Station  Working  Group,  MANPRINT  Working  Group,  and  System  Safety 
Working  Group  to  track  issues  until  satisfactorily  resolved. 

•  During  future  evaluations,  use  operational  pilots  as  test  subjects  instead  of  XPs.  XPs  have 
an  extraordinary  level  of  expertise  that  is  valuable  in  the  design  stage  of  a  program  and 
during  crew  station  working  groups;  however,  their  high  level  of  training  and  experience  is 
not  representative  of  the  intended  user  population,  and  using  them  during  HFEs  can  have 
an  adverse  impact  on  workload,  SA,  and  eye  tracker  results. 

•  Maximize  the  amount  of  “hands-on”  training  provided  to  the  test  participants  before  future 
evaluations.  Several  pilots  indicated  that  they  experienced  problems  retaining  the  location 
of  functions  during  the  LEUE.  This  training  shortfall  may  have  negatively  impacted 
workload  ratings. 

•  Future  work  should  include  a  formal  operational  evaluation  in  a  production  representative 
aircraft  or  simulator.  Because  of  the  limitations  associated  with  the  LEUE,  such  an 
evaluation  is  required  to  accurately  determine  the  workload  and  SA  associated  with  flying 
the  UH-60M  CAAS. 

•  Continue  to  assess  the  crew  station  during  future  simulations  and  tests  to  evaluate  pilot  and 
system  performance  and  assess  new  functionality  that  is  integrated  into  the  UH-60M 
design.  Data  from  the  workload,  SA,  and  SSQs,  plus  the  data  from  the  eye  tracker,  should 
be  collected  again  during  future  UH-60M  crew  station  evaluations.  This  procedural 
continuity  will  allow  direct  comparison  after  further  design  and  development  of  the 
UH-60M  crew  station  to  check  for  continued  improvements  in  workload,  SA,  and  PVI. 


26 


6.  References 


Crowley,  J.  S.  Simulator  sickness:  A  problem  for  army  aviation.  Aviation  Space  and 
Environmental  Medicine  1987,  58,  355-357. 

Department  of  the  Army  Headquarters.  TM  1-1520-237-10:  Operator’s  Manual  for  UH-60A, 
UH-60L,  EH-60A  Helicopters,  2004. 

Endsley,  M.  R.  Design  and  evaluation  for  situation  awareness  enhancement.  Proceedings  of  the 
Human  Factors  Society  32nd  Annual  Meeting,  Vol.l,  92-101,  1988. 

Endsley,  M.  R.  Situation  awareness  analysis  and  measurement.  Mahwah,  NJ:  Lawrence 
Erlbaum  Associates,  2000. 

Kennedy,  R.  S.;  Lane,  N.  E.;  Berbaum,  K.  S.;  Lilienthal,  M.  G.  Simulator  sickness 

questionnaire:  An  enhanced  method  for  quantifying  simulator  sickness.  International 
Journal  of  Aviation  Psychology  1993,  3,  203-220. 

Kennedy,  R.  S.;  Lilienthal,  M.  G.;  Berbaum,  B.  A.;  Balzley,  B.  A.;  McCauley,  M.  E.  Simulator 
sickness  in  U.S.  navy  flight  simulators.  Aviation  Space  and  Environmental  Medicine  1989, 
60,  10-16. 

O’Brien,  T.  G.;  Charlton,  S.  G.  Handbook  of  Human  Factors  Testing  and  Evaluation,  Lawrence 
Erlbaum  Associates,  Mahweh,  New  Jersey,  1996. 

Roscoe,  A.  H.  The  airline  pilots  view  of fight  deck  workload:  A  preliminary  study  using  a 
questionnaire.  Technical  Memorandum  No.  FS  (B)  465.  Bedford,  UK:  Royal  Aircraft 
Establishment.  ADA1 16314,  1985. 

Roscoe,  A.  H.;  Ellis,  G.  A.  A  Subjective  Rating  Scale  For  Assessing  Pilot  Workload  In  Flight:  A 
Decade  Of  Practical  Use.  Royal  Aerospace  Establishment,  Bedford,  UK,  1990. 

Taylor,  R.  M.  Situational  awareness  rating  technique  (SART):  The  development  of  a  tool  for 
aircrew  systems  design.  Proceedings  of  the  NATO  Advisory  Group  on  Aerospace  Research 
and  Development  (AGARD)  Symposium  on  Situational  Awareness  in  Aerospace  Operations . 
AGARD-CP-478,  3/1  -  3/17).  Neuilly  Sur  Seine,  France:  NATO  -  AGARD.  1989. 


27 


Intentionally  left  blank 


28 


Appendix  A.  Bedford  Workload  Rating  Scale 


1.  PIN _  2.  Date  (DD/MMM/YY): _ / _ /04 

3.  Mission  ID  number  _ 

4.  Right  Seat _  Left  Seat  _  (Check  one) 

Workload 

5.  Rate  the  workload  for  the  Flight  and  Mission  Tasks  you  performed  during  the  mission.  Use 
the  scale  provided  on  the  last  page  of  this  questionnaire.  Place  the  workload  rating  in  the  blank 
next  to  each  Flight  and  Mission  Task.  If  you  did  not  perform  a  task  during  the  mission  that  you 
just  completed,  place  an  X  in  the  non-applicable  (N/A)  column. 


Task 

No. 

Flight  and  Mission  Tasks 

UH-60M 

CAAS 

Workload 

N/A 

1024 

Perform  Before  Starting  Engines  through  Before  Leaving 
Helicopter  Checks 

1026 

Maintain  Airspace  Surveillance 

1028 

Perform  Hover  Power  Check 

1032 

Perform  Radio  Communications  Procedures 

1034 

Perform  Ground  Taxi 

1038 

Perform  Hovering  Flight 

1040 

Perform  VMC  Takeoff 

1044 

Navigate  by  Pilotage  and  Dead  Reckoning 

1046 

Perform  Electronically  Aided  Navigation 

1048 

Perform  Fuel  Management  Procedures 

1052 

Perform  VMC  Flight  Maneuvers 

1054 

Select  Landing  Zone/Pickup  Zone/Holding  Area 

1058 

Perform  VMC  Approach 

1070 

Perform  Emergency  Procedures 

1142 

Perform  Digital  Communications 

1164 

Perform  Instrument  Maneuvers 

1169 

Perform  Flight  Director  Operations 

1170 

Perform  Instrument  Takeoff 
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Task 

No. 

Flight  and  Mission  Tasks  (cont’d) 

UH-60M 

CAAS 

Workload 

N/A 

1172 

Perform  Radio  Navigation 

1178 

Perform  Precision  Approach 

1182 

Perform  Unusual  Attitude  Recovery 

1184 

Respond  to  Inadvertent  IMC 

1253 

Operate  Central  Display  Unit  (CDU) 

1254 

Operate  Multifunction  Display  (MFD) 

1260 

Operate  Digital  Map 

2024 

Perform  Terrain  Flight  Navigation 

2026 

Perform  Terrain  Flight 

2032 

Negotiate  Wire  Obstacles 

If  you  gave  a  workload  rating  of  ‘6’  or  higher  for  any  task  in  the  UH-60M  CAAS,  explain  why 
the  workload  was  high  for  the  task. 
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In  the  mission  you  just  flew,  list  any  flight  and/or  mission  tasks  in  the  UH-60M  CAAS  that  you 
had  to  ask  your  crew  member  to  accomplish  because  your  workload  was  too  high: 
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Workload  Description  “Rating” 


YES 


— * 

Wortdoad  insignificant 

1 

Workload  low 

2 

— ► 

Enough  spare  capacity  for  all 
desirable  additional  tasks 

3 

Was  wa&kM 

saiie&steisx 

wit.ljgjit  reduction  in 
sp-^  (workload)  capacity? 


NO 


Task  abandoned.  Pilot  unable  to 
apply  sufficient  effort 


Insufficient  spare  capacity  for  easy 
attention  to  additional  tasks 

4 

Reduced  spare  capacity.  Additional 
tasks  cannot  be  given  the  desired 
amount  of  attention 

5 

Little  spare  capacity:  level  of 
allows  little  attention  to  additional 
tasks 

6 

Very  little  spare  capacity,  but 
maintenance  of  effort  in  the  primary 
tasks  not  in  question 

7 

Very  lugihvrarkbadvath  almost  no 

- ► 

Difficulty  in 

maintaining  level  of  effort 

8 

Extremely  high  workload.  No  spare 
capacity.  Serious  doubts  as  to  ability 
to  maintain  level  of  effort 

9 

10 
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Appendix  B.  Situational  Awareness  Rating  Technique 


Pin  # _  Date  (DD/MM/YY): _ / _ /  04 

Mission  ID  Number:  _ 

Right  Seat _  Left  Seat  _  (Check  one) 

Situation  Awareness 


SA1.  Situation  Awareness  is  defined  as  “timely  knowledge  of  what  is  happening  as  you  perform 
your  right  or  left  seat  tasks  during  the  mission.” 


Situation  Awareness  Rating  Technique  (SART) 

DEMAND 

Instability  of  Situation 

Likeliness  of  situation  to  change  suddenly. 

Variability  of  Situation 

Number  of  variables  which  require  your  attention 

Complexity  of  Situation 

Degree  of  complication  (number  of  closely  connected  parts)  of  the 
situation 

SUPPLY 

Arousal 

Degree  to  which  you  are  ready  for  activity;  ability  to  anticipate  and 
keep  up  with  the  flow  of  events 

Spare  Mental  Capacity 

Amount  of  mental  ability  available  to  apply  to  new  tasks 

Concentration 

Degree  to  which  your  thoughts  are  brought  to  bear  on  the  situation; 
degree  to  which  you  focused  on  important  elements  and  events 

Division  of  Attention 

Ability  to  divide  your  attention  among  several  key  issues  during 
the  mission;  ability  to  concern  yourself  with  many  aspects  of 
current  and  future  events  simultaneously 

UNDERSTANDING 

Information  Quantity 

Amount  of  knowledge  received  and  understood 

Information  Quality 

Degree  of  goodness  or  value  of  knowledge  communicated 

Familiarity 

Degree  of  acquaintance  with  the  situation 
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For  the  mission  that  you  just  completed  in  the  UH-60M  CAAS,  rate  the  level  of  each  component 
of  situation  awareness  that  you  had.  Circle  the  appropriate  number  for  each  component  of 
situation  awareness  (e.g.,  complexity  of  situation). 


Instability  of  situation:  Low 
Variability  of  situation:  Low 
Complexity  of  situation:  Low 


DEMAND 

1 - 2 - 3 - 4 - 5 - 6 - 7  High 

1 - 2 - 3 - 4 - 5 - 6 - 7  High 

1 - 2 - 3 - 4 - 5 - 6 - 7  High 


SUPPLY 


Arousal: 

Low 

Low 

1—  ■ 

_ 2 _ 

...3.... 

__3 _ 

- 4... 

4 

.—6— 

_ 6 _ 

- 7 

7 

High 

High 

Concentration: 

Low 

1 .... 

_ 2 _ 

_ 3 _ 

4  - 

5 

7 

High 

_ _ _ 

Division  of  attention: 

Low 

1  — 

_ 2 _ 

..3..... 

. — 4.... 

_ 5 _ 

—6— 

- 7 

High 

UNDERSTANDING 

Information  quantity: 

Low 

1 _ 2 _ 3 _ 4 _ 

_ 5 _ 

_ g _ 

7 

High 

Information  quality: 

Low 

1 _ 2 _ 3 _ 4 _ 

_ 5 _ 

_ g _ 

7 

High 

Familiarity: 

Low 

1 _ 2 _ 3 _ 4 _ 

_ 5 _ 

. — 6-— 

. — 7 

High 
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Describe  any  instances  when  you  feel  you  had  low  situational  awareness  during  the  mission: 
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Intentionally  left  blank 
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Appendix  C.  PVI  Questionnaire 


1.  PIN _  2.  Date  (DD/MMM/YY): _ / _ /04 

3.  Mission  ID  number  _ 

4.  Right  Seat _  Left  Seat  _  (Check  one) 

PVI.  The  following  table  lists  the  functional  components  of  the  UH-60M  CAAS  crew  station. 
For  each  functional  component,  indicate  whether  or  not  you  experience  a  problem  using  the 
component  in  a  quick  and  efficient  manner  during  the  mission  you  just  completed.  Circle 
“Yes”  if  you  experience  one  or  more  problems.  Check  “No”  if  you  did  not  experience  any 
problems.  Circle  “Not  Used”  if  you  did  not  use  the  functional  component  during  the  mission 
you  just  completed. 

•  Multifunction  Displays  (MFD) 


o  Vertical  Situation  Display  (VSD) 

Yes  _ 

1 

No  _ 

0 

Not  Used  _ 

2 

o  VSD  Hover  (VSDH) 

Yes  _ 

1 

No  _ 

0 

Not  Used  _ 

2 

o  Horizontal  Situation  Display  (HSD)Yes_ 

1 

No  _ 

0 

Not  Used  _ 

2 

o  HSD  Hover  (HSDH) 

Yes  _ 

1 

No  _ 

0 

Not  Used  _ 

2 

o  Digital  Map  Display  (DMS) 

Yes  _ 

1 

No  _ 

0 

Not  Used  _ 

2 

o  Warning,  Caution,  Advisory 
Display  (WCA) 

Yes  _ 

1 

No  _ 

0 

Not  Used  _ 

2 

o  Engine  Instrument  Caution 
Advisory  System  (EICAS) 

Yes  _ 

1 

No  _ 

0 

Not  Used  _ 

2 

o  Tactical  Data  Link  Messaging 
(TDL) 

Yes  _ 

1 

No  _ 

0 

Not  Used  _ 

2 

Control  Display  Unit  (CDU) 

o  Initializing  CDU 

Yes  _ 

1 

No  _ 

0 

Not  Used  _ 

2 

o  Typing  TDL  Messages  on  CDU 

Yes  _ 

1 

No  _ 

0 

Not  Used  _ 

2 

o  Managing  GPS  /  FP 

Yes  _ 

1 

No  _ 

0 

Not  Used  _ 

2 

o  Managing  COM,  NAV,  IFF  (CNI) 

Yes  _ 

1 

No  _ 

0 

Not  Used  _ 

2 

Multifunction  Knob  (MFK) 

Yes  _ 

1 

No  _ 

0 

Not  Used  _ 

2 
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If  you  answered  “Yes”  to  any  of  the  questions,  please  describe  a)  the  problems  you 
experienced,  b)  how  much  the  problems  degraded  your  performance,  and  c)  any 
recommendation  you  have  for  improving  the  design  of  the  various  functional  components. 
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PV2.  Rate  the  ease  of  use  for  the  following  menu  screens  on  the  MFD’s  (i.e.  how  quickly 
could  you  navigate  through  the  menus  and  remember  the  steps  required  to  get  to  each 
page). 

Vertical  Situation  Display  /  Horizontal  Situation  Display  (VSD/HSD)  (Circle  one) 


1 

2 

3 

4 

5 

Very 

Easy 

Somewhat 

Easy 

Borderline 

Somewhat 

Difficult 

Very 

Difficult 

Digital  Map  System  (DMS) 

(Circle  one) 

1 

2 

3 

4 

5 

Very 

Easy 

Somewhat 

Easy 

Borderline 

Somewhat 

Difficult 

Very 

Difficult 

Control  Display  Unit  (CDU) 

(Circle  one) 

1 

2 

3 

4 

5 

Very 

Easy 

Somewhat 

Easy 

Borderline 

Somewhat 

Difficult 

Very 

Difficult 

Engine  Instrument  Caution  Advisory  System  (EICAS)  (Circle  one) 

1 

2 

3 

4 

5 

Very 

Easy 

Somewhat 

Easy 

Borderline 

Somewhat 

Difficult 

Very 

Difficult 

Tactical  Data  Link  Messaging  (TDL) 

1 

2 

3 

4 

5 

Very 

Easy 

Somewhat 

Easy 

Borderline 

Somewhat 

Difficult 

Very 

Difficult 

Flight  Director  (FD) 

1 

2 

3 

4 

5 

Very 

Easy 

Somewhat 

Easy 

Borderline 

Somewhat 

Difficult 

Very 

Difficult 
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If  you  answered  “Somewhat  Difficult”,  or  “Very  Difficult”  to  any  of  the  questions,  list  the 
component  and  why  navigation  was  difficult  (e.g.,  ‘navigating  the  menu  system  on  the  FMS 
was  a  slow  process  due  to  having  to  page  through  several  screen  displays’). 


PV3.  Please  answer  the  following  questions  regarding  the  Multifunction  Control  Unit 
(MFCU): 

PV3-1.  Did  the  functionality  of  the  directional  control  and  switches  on  the  MFCU  perform  the 
actions  you  expected? 

Yes  1  No  0  Not  Used  2 

PV3-2.  Was  the  sensitivity  of  the  directional  control  appropriate? 

Yes  1  No  0  Not  Used  2 

PV3-3.  Did  you  experience  abnormal  hand  discomfort  while  using  the  MFCU? 

Yes  1  No  0  Not  Used  2 

If  you  experienced  any  problems  with  the  MFCU,  please  describe  the  problems  in  as  much 
detail  as  you  can  and  make  recommendations  to  correct  the  problems. 
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PV4.  Did  you  experience  abnormal  fatigue  or  have  difficulty  using  any  of  the  switches  on 
the  collective  or  cyclic  grips? 


Collective  Grip 

Yes _ 

1 

No _ 

0 

Cyclic  Grip 

Yes 

1 

No 

0 

If  you  answered  “Yes”  for  either  flight  control,  please  list  which  flight  control  and  switch(es), 
and  the  problems  you  experienced  (e.g.,  confused  two  switches  due  to  similar  shape, 
switch  too  hard  to  reach). 


PV5.  Was  there  any  symbology  depicted  on  the  following  displays/pages  that  was  difficult 
to  quickly  and  easily  understand,  cluttered,  or  otherwise  difficult  to  use? 


Vertical  Situation  Display  (VSD) 

Yes _ 

1 

No 

VSD  Hover  (VSDH) 

Yes _ 

1 

No 

Horizontal  Situation  Display  (HSD)Yes_ 

1 

No 

HSD  Hover  (HSDH) 

Yes _ 

1 

No 

EICAS 

Yes _ 

1 

No 

Digital  Map  System  (DMS) 

Yes _ 

1 

No 

Tactical  Data  Link  Message  (TDL)  Yes  1  No _ 0 
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If  you  answered  “Yes”  to  any  of  the  questions,  please  describe  a)  the  display/page,  b)  the 
symbology  that  was  difficult  to  understand,  c)  how  the  symbology  may  have  degraded  your 
performance,  and  d)  any  recommendation  you  have  for  improving  the  design  of  the  various 
functional  components. 


PV6.  For  the  TDL  reports  that  you  sent,  how  would  you  rate  the  ease/difficulty  of  sending 
the  following  reports: 

Position  Report 


1 

2 

3 

4 

5 

Very 

Easy 

Somewhat 

Easy 

Borderline 

Somewhat 

Difficult 

Very 

Difficult 

Free  Text  Message 

1 

2 

3 

4 

5 

Very 

Easy 

Somewhat 

Easy 

Borderline 

Somewhat 

Difficult 

Very 

Difficult 

Observation  Report 

1 

2 

3 

4 

5 

Very 

Easy 

Somewhat 

Easy 

Borderline 

Somewhat 

Difficult 

Very 

Difficult 

If  you  answered  “Somewhat  Difficult”,  or  “Very  Difficult”,  please  indicate  which  type  of 
message  you  sent,  the  exact  difficulties  you  encountered,  and  any  recommendations  to 
alleviate  the  problem. 
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PV7.  How  would  you  rate  your  ability  to  detect  the  following  occurrences  based  on  the 


characteristics  of  the  flight  displays? 

Incoming  TDL  Message  (MFD) 

1  2  3 

4 

5 

Very 

Somewhat 

Borderline 

Somewhat 

Very 

Easy 

Easy 

Difficult 

Difficult 

Caution  /  Advisory  (MFD) 

1 

2 

3 

4 

5 

Very 

Somewhat 

Borderline 

Somewhat 

Very 

Easy 

Easy 

Difficult 

Difficult 

Warning  (Master  Warning  Panel) 

1 

2 

3 

4 

5 

Very 

Somewhat 

Borderline 

Somewhat 

Very 

Easy 

Easy 

Difficult 

Difficult 

Entry  into  Operational  Limits  (per  Chp  5) 

1 

2 

3 

4 

5 

Very 

Somewhat 

Borderline 

Somewhat 

Very 

Easy 

Easy 

Difficult 

Difficult 

Low  Fuel  (MFD) 

1 

2 

3 

4 

5 

Very 

Somewhat 

Borderline 

Somewhat 

Very 

Easy 

Easy 

Difficult 

Difficult 
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If  you  answered  “Somewhat  Difficult”,  or  “Very  Difficult”,  please  indicate  which  annunciation 
you  had  difficulty  detecting,  why  you  may  have  had  difficulty  detecting  it,  and  any 
recommendations  to  make  the  annunciation  more  easily  detectable. 
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Intentionally  left  blank 
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Appendix  D.  Simulator  Sickness  Questionnaire 


1.  PIN  #: _  2.  Date  (DD/MMM/Y Y) : _ - _ -  04 

3.  Mission  ID  Number:  _ 

4.  Seat  you  will  fly  from:  Right  Seat _  Left  Seat  _  (Check  one) 

5.  Please  indicate  the  severity  of  symptoms  that  apply  to  you  right  now  by  circling  the  appropriate  word. 


Symptom 

0 

1 

2 

3 

a. 

General  discomfort 

None 

Slight 

Moderate 

Severe 

b. 

Fatigue 

None 

Slight 

Moderate 

Severe 

c. 

Headache 

None 

Slight 

Moderate 

Severe 

d. 

Eyestrain 

None 

Slight 

Moderate 

Severe 

e. 

Difficulty  focusing 

None 

Slight 

Moderate 

Severe 

f. 

Increased  salivation 

None 

Slight 

Moderate 

Severe 

g- 

Sweating 

None 

Slight 

Moderate 

Severe 

h. 

Nausea 

None 

Slight 

Moderate 

Severe 

i. 

Difficulty  concentrating 

None 

Slight 

Moderate 

Severe 

j- 

Fullness  of  head 

None 

Slight 

Moderate 

Severe 

k. 

Blurred  vision 

None 

Slight 

Moderate 

Severe 

1.  Dizzy  (eyes  open) 

None 

Slight 

Moderate 

Severe 

m. 

.  Dizzy  (eyes  closed) 

None 

Slight 

Moderate 

Severe 

n. 

* 

Vertigo 

None 

Slight 

Moderate 

Severe 

0. 

** 

Stomach  awareness 

None 

Slight 

Moderate 

Severe 

P- 

Burping 

None 

Slight 

Moderate 

Severe 

Vertigo  is  a  loss  of  orientation  with  respect  to  vertical  upright. 


Stomach  awareness  is  a  feeling  of  discomfort  just  short  of  nausea. 


6.  Are  you  in  your  usual  state  of  health  and  fitness? 

YES 

NO 

7a.  Have  you  been  ill  in  the  past  week? 

YES 

NO 

b.  If  yes,  are  you  fully  recovered? 

YES 

NO 

N/A 
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1.  PIN#: 


2.  Date  (DD/MMM/YY): 


-04 


3.  Mission  ID  Number:  _ 

4.  Seat  you  flew  from:  Right  Seat _  Left  Seat  _  (Check  one) 

5.  Please  indicate  the  severity  of  symptoms  that  apply  to  you  right  now  by  circling  the 
appropriate  word. 


Symptom 

0 

1 

2 

3 

a.  General  discomfort 

None 

Slight 

Moderate 

Severe 

b.  Fatigue 

None 

Slight 

Moderate 

Severe 

c.  Headache 

None 

Slight 

Moderate 

Severe 

d.  Eyestrain 

None 

Slight 

Moderate 

Severe 

e.  Difficulty  focusing 

None 

Slight 

Moderate 

Severe 

f.  Increased  salivation 

None 

Slight 

Moderate 

Severe 

g.  Sweating 

None 

Slight 

Moderate 

Severe 

h.  Nausea 

None 

Slight 

Moderate 

Severe 

i.  Difficulty  concentrating 

None 

Slight 

Moderate 

Severe 

j.  Fullness  of  head 

None 

Slight 

Moderate 

Severe 

k.  Blurred  vision 

None 

Slight 

Moderate 

Severe 

1.  Dizzy  (eyes  open) 

None 

Slight 

Moderate 

Severe 

m.  Dizzy  (eyes  closed) 

None 

Slight 

Moderate 

Severe 

n.  Vertigo* 

None 

Slight 

Moderate 

Severe 

o.  Stomach  awareness** 

None 

Slight 

Moderate 

Severe 

p.  Burping 

None 

Slight 

Moderate 

Severe 

9J 

Vertigo  is  a  loss  of  orientation  with  respect  to  vertical  upright. 
Stomach  awareness  is  a  feeling  of  discomfort  just  short  of  nausea. 
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Appendix  E.  Tactical  Steering  Committee  Questionnaire 


Pin: 


Mission  Trial _  Date  (DD/MMM/YY): _ / _ /04 

TSCWL1.  Place  the  workload  rating  in  the  blank  next  to  each  crewmember  using 
the  rating  scale  on  the  next  page. 


Crew  members 

Overall  Workload  Rating  For  This 
Mission 

Left  Seat 

Right  Seat 

If  you  assigned  a  workload  rating  of  ‘6’  or  higher  for  either  crewmember,  explain  why: 


TSCWL2.  Rate  the  effectiveness  of  aircrew  coordination  as  defined  by  the  USAAVNC  Aircrew 
Coordination  ETP  and  TC  1-210. 

1  2  3  4  5 


Excellent  Good  Average  Needs  Improvement  Unacceptable 
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PIN 


Date  (DD/MMM/YY) 


/ 


/  0  4 


Mission  ID  number 


TSC  SITUATION  AWARENESS  RATING  SCALE 


UH-60M  CAAS 

Circle  One 

Crew  was  consistently  aware  of  all  entities  on  the  battlefield  as  well  as  the  status  of  their 

aircraft 

1 

Crew  was  aware  of  the  battlefield  and  their  own  ship  with  minor  or  insignificant  variation 

between  perception  and  reality. 

2 

Crew  was  aware  of  the  battlefield  and  their  own  ship.  Variation  between  reality  and  perception 
did  not  significantly  impact  mission  success. 

3 

SA  needs  improvement.  Lack  of  SA  had  some  negative  effect  on  the  success  of  the  mission. 

4 

Lack  of  SA  caused  mission  failure. 

5 

Describe  any  problems  that  aircrews  had  with  situation  awareness. 
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PIN 


Date  (DD/MMM/YY): 


/ 


/  0  4 


Mission  ID  number  _ 

TSC  MISSION  SUCCESS  QUESTIONNAIRE 

TSC  MSI.  Did  the  UH-60M  CAAS  crew  complete  their  mission  objectives? 

Yes  1  No  0 

If  no,  why  weren’t  the  mission  objectives  completed? 


TSC  MS2.  Was  the  mission  successful? 

Yes  1  No  _ 0 

If  no,  what  caused  the  mission  to  fail? 
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Intentionally  left  blank 
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L/l 

u> 


ATM  Task  Description 


1024 

Perform  Before  Starting  Engines  through  Before 
Leaving  Helicopter  Checks 

1026 

Maintain  Airspace  Surveillance 

1028 

Perform  Hover  Power  Check 

1032 

Perform  Radio  Communications  Procedures 

1034 

Perform  Ground  Taxi 

1038 

Perform  Hovering  Flight 

1040 

Perform  VMC  Takeoff 

1044 

Navigate  by  Pilotage  and  Dead  Reckoning 

1046 

Perform  Electronically  Aided  Navigation 

1048 

Perform  Fuel  Management  Procedures 

1052 

Perform  VMC  Flight  Maneuvers 

1054 

Select  Landing  Zone/Pickup  Zone/Holding  Area 

1058 

Perform  VMC  Approach 

1070 

Perform  Emergency  Procedures 

1142 

Perform  Digital  Communications 

1164 

Perform  Instrument  Maneuvers 

1169 

Perform  Flight  Director  Operations 

1170 

Perform  Instrument  Takeoff 

1172 

Perform  Radio  Navigation 

1178 

Perform  Precision  Approach 

1182 

Perform  Unusual  Attitude  Recovery 

1184 

Respond  to  Inadvertent  IMC 

1253 

Operate  Central  Display  Unit  (CDU) 

1254 

Operate  Multifunction  Display  (MFD) 

1260 

Operate  Digital  Map 

2024 

Perform  Terrain  Flight  Navigation 

2026 

Perform  Terrain  Flight 

2032 

Negotiate  Wire  Obstacles 

2  3  4 

loo  T67 

TOO  T67  TOO 

467  L50  L33 

T67  T67  L67 

T50  LOO  L50 

T50  ZOO  ZOO 

T50  ZOO  230 

333  230  T67 

400  230  TOO 

433  230  T67 

450  TOO 

230  LOO  T50 

TOO  TOO 

— 

TOO 

450  TOO  230 

433  TOO  T67 

TOO  TOO 

400  TOO  T67 

400  TOO  TOO 

TOO 

TOO  400  TOO 

5.67  2.67  3.67 

4.67  3.00  3.33 

433  T67  33J 

TOO  TOO  400 

TOO  400 

230  TOO 


Intentionally  left  blank 
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Appendix  G.  Pilot  Workload  Comments 


If  you  gave  a  workload  rating  of  6  or  higher  on  the  UH-60M  CAAS,  explain  why  the  workload 
was  high  for  the  task. 

Task  (1026) 

•  The  heads-down  nature  of  the  CDU  brought  the  flying  pilot  inside  for  too  long. 

•  Resetting  flight  director  commands  was  too  heads-down. 

Task  (1028) 

•  Had  to  bring  up  instrument  page  and  increase  scan  to  verify  all  engine  parameters  were 
okay.  Would  like  to  see  more  on  the  power  pod. 

•  Had  to  cycle  through  instrument  pages  to  get  all  the  required  aircraft  information. 

•  Cockpit  is  too  heads-down.  Need  fuel,  Kollsman  window,  Nr,  TGT,  and  Ng  pop-ups  on 
PFD.  Need  easy  way  of  changing  flight  director  settings.  Need  ONE  all-inclusive 
EICAS  page. 

Task  (1032) 

•  Collective  radio  control  was  easy  to  use.  Would  like  to  be  able  to  see  what  radio  the 
other  pilot  had  selected  on  the  MFD. 

•  Would  like  to  see  other  pilots  selected  radio  on  MFD. 

Task  (1044) 

•  Heads-down  nature  of  cockpit  brings  you  inside  too  much. 

•  Heads-down  cockpit.  No  heading  bug  without  flight  director. 

Task  (1046) 

•  Had  to  scan  other  pilots  MFD  for  exact  position  since  my  map  display  didn’t  have  an 
aircraft  symbol. 

•  Moving  map  is  good.  Would  like  aircraft  symbol  on  all  digmap  pages.  Recommend 
making  the  ACP,  ADF,  VOR,  and  TACAN  needles  different  colors. 

•  Need  easier,  heads-up  way  of  adjusting  heading,  course,  and  flight  director  settings. 

Task  (1048) 

•  Had  to  pull  up  a  fuel  display  while  on  controls  to  monitor  fuel. 

•  Need  fuel  display  on  PFD. 

•  No  fuel  on  PFD. 

Task  (1052) 

•  Angle  of  bank  pointer  moved  in  counter  intuitive  direction  causing  me  to  steepen  turns 
instead  of  rolling  wings  level. 

•  Heads-down  cockpit.  No  heading  bug  without  flight  director. 

Task  (1058) 


55 


•  If  heavy  or  coming  in  “Hot”,  Ng  and  TGT  pop-ups  on  PFD  would  be  helpful. 

Task  (1142) 

•  Digital  communications  took  from  1 1  to  20+  button  pushes  to  send  a  simple  position 
report.  At  least  30  seconds  inside  the  cockpit  to  perfonn  this  task. 

Task  (1164) 

•  The  heads-down  nature  of  the  heading  and  course  bugs  made  setting  up  the  cockpit  for 
the  flying  pilot  difficult. 

•  High  workload  for  basic  NAV  caused  by  cumbersome  input  devices. 

•  Need  easier  way  of  changing  FD  settings. 

Task  (1169) 

•  The  heads-down  nature  of  the  heading  and  course  bugs  made  setting  up  the  cockpit  for 
the  flying  pilot  difficult. 

•  Need  easier  way  of  changing  FD  settings. 

Task  (1172) 

•  Changing  course-to  was  heads-down. 

Task  (1176) 

•  Changing  course-to  was  heads-down. 

Task  (1184) 

•  This  task-saturated  event  is  turned  into  a  task-unmanageable  event  by  the  introduction  of 
current  CAAS.  What  was  my  altimeter  setting?  We  leveled  off  and  still  did  not  know.  I 
did  not  have  time  to  tune  a  VOR  prior  to  level  off,  therefore  giving  a  position  report  was 
somewhat  inaccurate  (the  map  was  no  help).  Putting  up  a  heading  bug  took  too  long 
because  it  was  not  on  a  main  page.  Setting  altimeter  was  too  time  consuming  and 
difficult. 

•  Finding  proper  overlay  to  quickly  bring  up  bearing  pointer  and  CDI  is  difficult. 

Task  (1253) 

•  Three  to  four  button  pushes  to  just  begin  to  enter  altimeter  setting.  Entering  data  with  the 
MKF  was  cumbersome  and  time  consuming  when  used  with  the  CDU.  Too  many  button 
pushes  to  enter  a  course  for  the  CDI. 

•  Menu  structure  has  too  many  layers,  information  is  difficult  to  find,  too  much  heads- 
down. 

•  Was  too  difficult  to  accomplish  the  basic  tasks  such  as  entering  altimeter,  course  set 
display,  and  heading  settings  while  flying  instruments. 

•  Menu  structure  non-intuitive  and  too  deep.  Should  be  able  to  set-up  FD  commands 
(altitude,  airspeed,  heading)  without  engaging  the  flight  director.  Overlay  and  zoom 
menu  structure  confusing.  Bezel  keys  should  indicate  where  I’ll  go  if  I  select  the  button, 
not  where  I’m  at  now. 

•  Data  entry  takes  way  too  long.  Need  more  hotkeys.  Primary  finction  are  buried  too  deep 
in  pages. 

•  Too  many  sub-menus.  Menu  structure  not  intuitive. 

Task  (1254) 
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•  Absolutely  must  be  an  altimeter/Kollsman  window  on  the  MFD.  Absolutely  must  be  able 
to  put  any  screen  in  any  position  on  the  MFD’s. 

•  PFD  doesn’t  display  enough  infonnation,  requiring  frequent  page  switching. 

•  Not  enough  basic/critical  data  displayed  on  MFD.  (e.g.,  altimeter  setting,  fuel)  Need 
more  hot  keys  and  top-level  functions.  Relable  MFD  key  to  show  the  next  function,  not 
the  current  function.  Add  knobs  and  buttons  to  compliment  MFD.  It  would  have  helped 
on  this  mission. 

•  Setting  up  split  screen  is  non-intuitive.  Too  many  flight  director  and  overlay  pages. 

Task  (1260) 

•  All  digital  map  displays  need  aircraft  symbol. 

•  Operating  the  flight  director  required  too  many  button  pushes.  For  example,  selecting 
heading  required  selecting  FD  from  the  MFD,  then  HDG  on  the  left  sike  LSK,  then  FD 
on  CDU,  either  entering  the  heading  manually  or  using  the  MFK  (which  I  did  not  use). 
This  is  way  too  much  workload  to  get  a  heading  indication. 

•  Set-up  difficult. 

Additional  Comments 

•  Operating  the  flight  director  required  too  many  button  pushes.  For  example,  selecting 
heading  required  selecting  FD  from  the  MFD,  then  HDG  on  the  left  sike  LSK,  then  FD 
on  CDU,  either  entering  the  heading  manually  or  using  the  MFK  (which  I  did  not  use). 
This  is  way  too  much  workload  to  get  a  heading  indication. 

•  Even  though  I  didn’t  have  any  workload  ratings  higher  than  6,  when  trying  to  perform 
multiple  tasks  that  score  3  or  4,  the  workload  becomes  high.  There  are  cases  where 
selecting  the  correct  overlay  on  the  HSD  was  difficult.  Each  pilot  may  have  a  difficult 
time  determining  what  the  other  pilot  has  displayed  in  terms  of  bearing  pointers  and  CDI 
(the  NAV  source,  such  as  CDI-VOR,  CDI-ADF).  This  may  show  conflicting  infonnation 
between  Left  and  Right  pilot  displays  and  could  lead  to  disorientation. 

•  Overall,  workload  is  too  high  in  setting  up,  using,  and  changing  MFDs,  flight  director 
modes,  etc.  Overflying  an  exact  waypoint  is  difficult  because  the  flying  pilot  cannot  have 
a  PFD,  digmap,  and  systems  displayed  simultaneously.  The  CDI  and  bearing  pointer 
auto  turn  prior  to  ACP  and  did  not  see  a  way  to  select  manual  feature  in  this  simulation. 

•  If  the  MFDs  were  designed  better  OR  there  was  an  auto  paging  when  needed  (e.g.,  low 
oil  pressure,  etc)  then  that  would  be  no  factor.  As  it  is,  you  must  monitor  to  ensure  you 
don’t  exceed  Ng,  etc. 


In  the  mission  you  flew,  list  any  tasks  in  the  UH-60M  CAAS  that  you  had  to  ask  your  crew 
member  to  accomplish  because  your  workload  was  too  high. 

•  The  pilot  on-the-controls  made  some  radio  calls  to  reduce  pilot  not-on-the-controls 
workload.  For  this  first  flight,  the  pilot-no t-on-the-controls  could  perform  all  functions 
but  with  increased  difficulty. 

•  Had  to  have  the  other  pilot  adjust  my  heading  and  course  bugs.  Too  heads-down  for  the 
flying  pilot  to  do. 

•  Set  altimeter  setting. 
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•  None,  however  I  did  have  to  ask  my  crewmember  about  how  to  perform  a  task  (i.e.  pull 
up  the  CDU  window  for  altimeter  setting. 

•  I  did  not  feel  comfortable  leaving  the  controls  to  search  for  and  fumble  through  pages  of 
menus  to  perform  basic  tasks  which  I  listed  before. 

•  I  did  not  ask  my  crewmember  to  do  any  specific  task  because  of  workload.  He  did 
change  heading  for  flight  director  and  airspeed  for  the  flight  director  so  I  could  get  flight 
director  indications. 

•  Set  altimeter,  time  to  next  ACP/CP,  landing  direction  into  LZ/PZ. 

•  Everything  that  didn’t  include  moving  the  flight  controls.  For  example,  changing 
altimeter  setting,  flight  director  settings,  heading  bug,  GDI,  etc. 

•  The  pilot  on-the-controls  has  to  set  his  own  airspeed  bug  because  I  was  inside  trying  to 
figure  out  whether  or  not  I  had  the  VOR  up  correctly. 

•  Describing  upcoming  waypoints,  since  I  had  my  MFDs  configured  to  fly  and  monitor 
systems. 

If  you  feel  the  addition  of  a  5th  MFD  would  have  helped  reduce  workload,  explain. 

•  On  this  flight  a  5  th  MFD  would  have  reduced  pilot  workload.  Fuel  and  instrumentation 
could  have  been  displayed  on  the  5  th  MFD  allowing  for  less  page  changes  and  lending 
more  space  on  the  primary  MFD’s  for  a  flight  data  oriented  page.  Alternatives  would  be 
to  add  total  fuel  and  fuel  burn  to  the  systems  page  and  also  allow  the  systems  page  to  be 
viewed  in  any  position.  Allow  for  a  Kollsman  window  to  be  displayed  somewhere  on  the 
MFD’s.  If  these  changes  cannot  be  made,  additional  display  space  is  required. 

•  Yes,  with  a  5th  MFD,  fuel,  altimeter  setting,  engine  and  transmission  instruments,  map, 
and  PFD  could  be  displayed  simultaneously. 

•  Here  it  would  have  allowed  me  to  have  system  instruments  and  fuel  status  displayed.  I 
would  not  have  to  switch  back  from  map  to  instruments  to  fuel. 

•  I  don’t  think  it  is  necessary  if  the  4  MFD’s  we  have  are  well  designed. 

•  Yes.  Allow  any  function  to  be  displayed  in  any  position.  Remove  torque  from  EIC AS 
and  insert  fuel  gauges  or  strips.  Create  more  space  on  MFD’s  or  CDU  for  hot  keys.  Put 
a  Kollsman  window  in  the  system  on  MFD.  If  not,  yes,  we  need  more  display  space. 

•  Yes,  could  have  better  monitored  aircraft  systems.  However,  I’d  rather  see  a  redesigned 
PFD  with  fuel,  Kollsman  window,  Ng;  and  an  EICAS  that  shows  engine  instrument, 
caution/advisory,  and  fuel  on  one  page. 

•  Yes.  Provided  more  space  for  hot  kays.  Provided  system  and  fuel  indication  or  other 
combinations  to  be  displayed  at  all  times  because  the  system  is  too  difficult  to  ma  ipulate 
under  high  stress  situations.  Therefore,  buy  not  having  to  select  different  pages  in  the 
current  software  you  decrease  workload  with  a  5th  MFD. 

•  Not  if  its  more  of  the  same.  Need  redesigned  PFD  and  EICAS. 
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Appendix  H.  Pilot  SA  Comments 


Describe  any  instances  when  you  feel  you  had  low  situation  awareness  during  the  mission. 

•  At  one  point,  inside  the  aircraft  too  long  accessing  course  and  may  have  made  contact 
with  the  ground! 

•  At  a  quick  glance,  I  reported  altitude  as  6000  feet  when  we  were  actually  at  600  feet. 
The  more  experienced  pilot  recognized  the  error  and  corrected  me.  Recommend  1st 
digit  be  larger  than  the  rest.  Currently  the  1 st  and  2nd  digits  are  larger  than  the  rest. 

•  I  couldn’t  display  all  the  infonnation  I  wanted  to  see  simultaneously.  I  had  the 
inboard  MFD  split-screen  with  the  map  on  top  and  instruments  on  bottom.  As  such,  I 
could  not  get  the  airplane  symbol  on  the  top  display  and  had  to  toggle  the  bottom  half 
between  systems  and  fuel. 

•  When  we  were  given  a  new  altimeter  setting,  my  copilot  had  to  come  inside  just  to 
see  if  it  was  the  same  as  the  last  setting. 

•  Could  not  set  heading  bug  to  next  course  leg  without  engaging  flight  director. 

•  Copilot  mis-interpreted  altitude  as  6000  feet  when  it  was  600  feet. 

•  When  using  the  digital  map,  the  aircraft  symbol  would  frequently  “get  lost”  in  the 
background.  The  current  course  leg  would  partially  “get  lost”  as  well.  This  is 
probably  due  mainly  to  lack  of  contrast  between  the  symbols  and  the  map  itself.  A 
change  in  size,  color,  or  shading  may  help. 

•  The  time  I  felt  low  SA  was  during  radio  calls  and  not  being  real  sure  about  how  I 
would  manually  switch  radio  frequencies  if  they  were  not  in  presets. 

•  Had  low  SA  of  my  current  status  due  to  the  inability  to  pull  up  engine  instruments, 
caution/advisory,  and  fuel  simultaneously. 

•  On  takeoff,  the  right  side  pilot  had  selected  CDI-ACP  and  I  had  CDI-VOR.  We  had 
different  indications,  obviously.  The  problem  was  that  I  cannot  tell  what  he  had 
selected  from  cross-cockpit,  causing  some  confusion  about  what  is  displayed  and 
exactly  where  we  were  navigating  to. 

•  Outbound  on  a  VOR  radial  (the  217  radial  at  Bike  Lake)  the  needle  (CDI)  indicated 
reverse  sensing.  It  should  be  directional. 

•  Had  difficulty  sensing  a  bank  on  the  VSD.  Usually  corrected  to  the  wrong  side. 

•  Difficult  for  me  to  know  what  information  the  lollipops  were  giving  me  on  the  VSD. 

I  asked  numerous  times  to  the  copilot  about  what  indication  that  symbol  was  giving 
me  (i.e.,  heading,  course,  etc.). 

•  Landing  direction  and  spot  at  PZ  not  coordinated  well. 

•  Wasn’t  comfortable  with  the  state  of  my  own  aircraft.  Need  an  all  inclusive  EICAS 
page. 

•  When  approaching  an  ACP,  the  CDI  bearing  pointers  auto  change  prior  to  overflight. 
(This  is  probably  selectable  in  the  real  CAAS)  The  flying  pilot  is  unable  to  have 
digital  map  with  overlay  AND  still  have  the  primary  flight  display  (PFD)  while 
monitoring  systems.  So  overflight  of  the  actual,  exact  point  that  defines  each  ACP  is 
difficult. 
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•  When  asked  to  tune  up  the  VOR,  but  did  not  get  the  VOR  BRG  pointer.  But  did  get 
the  GDI  - 1  did  not  know  if  GDI  indication  were  good  or  not.  (No  NAV  flag!) 

•  When  flying  along  and  spotted  the  enemy  vehicles,  I  did  not  know  how  to  store  that 
GPS  coordinate  in  a  quick  fashion  to  report  later. 

If  you  feel  the  addition  of  a  5th  MFD  would  have  helped  increase  situation 

awareness,  explain. 

•  On  this  flight,  4  MFD’s  lended  itself  more  suitable  than  5.  My  adaptation  to  the  flight 
deck  configuration  has  helped  in  the  manipulation  of  seeming  awkward  controls. 

•  5th  MFD  not  needed. 

•  I’d  rather  see  a  Kollsman  window  and  fuel  display  on  4  MFD’s.  Additionally,  need  a 
quick  way  of  setting  heading,  course,  altitude,  airspeed,  etc  without  going  heads- 
down  to  the  CDU.  If  the  MFD  displays  cannot  be  modified  then  a  5l  MFD  is 
definitely  required.  Need  to  be  able  to  display  bug  (i.e.  heading,  GDI,  etc)  without 
engaging  the  flight  director. 

•  I  don’t  think  so. 

•  Not  during  this  mission. 

•  Yes,  display  fuel  would  have  helped.  Altimeter  setting  on  a  5th  display  would  help. 
Otherwise,  design  these  issues  into  the  4  MFD  configuration. 

•  I  would  rather  see  a  well  design  PFD  with  fuel,  Kollsman  window,  and  NR  on  the 
PFD  and  a  redesigned  EICAS  with  engine  instruments,  caution/advisory,  and  fuel  on 
one  page.  If  the  pages  can’t  be  redesigned,  then  a  5th  MFD  will  be  required. 

•  Yes.  To  display  instruments  and  fuel  or  to  display  fuel  and  WCA  a  5th  MFD  is 
required.  We  must  have  more  space  unless  we  fix  the  issues  with  the  4  MFD 
configuration.  Even  “more  of  the  same”  is  still  displaying  more  readily  and 
immediate  available  infonnation.  A  5  th  display  will  reduce  pilot  workload  with  this 
software. 

•  Personal  opinion  -  Fix  the  data  input  issues,  safety  concerns,  and  data  interpretation 
issues  that  we  have  discussed.  Then  talk  about  using  4  MFD’s.  In  actuality,  this 
system  is  not  safe  as  is. 

•  Not  if  just  more  of  the  same.  We  need  a  redesigned  PFD  and  EICAS. 

•  Today  with  a  5th  MFD  I  would  have  displayed  instruments!  However,  I  feel  the  one 
page  redesigned  Engine/Systems  MFD  page  would  be  just  as  useful  (i.e,  ALL  stsrems 
&  fuel  on  one  single  page.  If  that  were  the  way  CAAS  was  - 1  could  do  with  4 
MFD’s. 
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Appendix  I.  TSC  Comments 


TSC  Workload  Comments: 

•  Several  button  pushes  were  required  to  perform  basic  copilot  functions  (e.g. 
systems  check,  before  take-off  check,  initiate  and  close-out  fuel  consumption 
check,  and  monitor  fuel  throughout  the  mission. 

TSC  SA  Comments: 

•  Copilot  had  difficulty  recalling  altimeter  setting  changes  (non-intuitive). 

•  Pilot  misinterpreted  turn  indicator  to  make  course  corrections.  Caused  by  non- 
intuitive  turn  indications. 

•  Copilot  management  of  flight  plan  was  not  implemented  to  it’s  fullest  capability. 
Didn’t  use  the  course  deviation  indicator  to  track  to  the  ACP. 

•  Pilot  on-the-controls  had  variations  in  altitude  and  airspeed  greater  than  ATM 
standard. 

•  Crew  did  not  have  full  CDU  functionality  to  capture  airspeed,  altitude,  and  course 
guidance  information  (i.e.,  did  not  use  CDU  to  capture  desired  airspeed  and 
altitude  to  receive  queues  for  airspeed  and  altitude). 

•  Pilot  on-the-controls  had  difficulty  interpreting  course  guidance  information  on 
the  VSD  attitude  indicator.  The  aviator  commented  the  display  was  too 
prominent  (e.g.,  bright-pink). 

•  JVMF  position  reporting  didn’t  allow  crew  to  execute  the  briefed  mission.  Could 
not  send  position  report  from  MFD.  Had  to  develop  work  around  to  use  the 
MFCU. 

•  IIMC  aviator  distracted  through  flight  plan  model  errors  that  caused  over 
controlling.  Eventually  the  aviator  was  able  to  establish  level  flight. 

•  Simulation  device  error  caused  aviator  workload  to  increase.  VOR  receiver  was 
not  operational.  Aviator  had  loss  of  situation  awareness  and  unable  to  recover 
from  inadvertent  IMC. 
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Intentionally  left  blank 
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Appendix  J.  Pilot  PVI  Summary  and  Comments 


PVl  Comments 


Did  you  experience  a  problem  using  the  VSDin 
a  quick  and  efficient  manner  during  the 
m  is  sion  you  just  com  pie  ted? 
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Did  you  experience  a  problem  using  the  VSDH  in 
a  quick  and  efficient  manner  during  the 
m  is  s  ions  you  just  com  pie  ted? 
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Did  you  experience  a  problem  using  the  HSD  in 
a  quick  and  efficient  manner  during  the 
m  is  s  ions  you  just  com  pie  ted? 
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Did  you  experience  a  problem  using  the  HSDH  in  a 
quick  and  efficient  manner  during  the  missions 
you  just  completed? 


Did  you  experience  a  problem  using  the  DMS  in  a 
quick  and  efficient  manner  during  the  missions 
you  just  completed? 
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Did  you  experience  a  problem  using  the  WCA 
displays  in  a  quick  and  efficient  manner  during  the 
missions  you  just  completed? 


64 


Did  you  experience  a  problem  using  the  EICAS  in  a 
quick  and  efficient  manner  during  the  missions  you 
just  completed? 


Did  you  experience  a  problem  using  the  TDL 
messages  in  a  quick  and  efficient  manner  during 
the  missions  you  just  completed? 


Did  you  experience  a  problem  initializing  the  CDU 
in  a  quick  and  efficient  manner  during  the 
missions  you  just  completed? 
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Did  you  experience  a  problem  typing  messages 
on  the  CDU  in  a  quick  and  efficient  manner  during 
the  missions  you  just  completed? 
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Did  you  experience  a  problem  managing 
the  GPS/flight  plan  in  a  quick  and  efficient 
manner  during  the  missions  you  just 
completed? 


Did  you  experience  a  problem  managing 
COM/NAV  in  a  quick  and  efficient  manner  during 
the  missions  you  just  completed? 


Can’t  set-up  for  next  part  of  flight  plan  without  changing  information  on  pilots  display. 
CDU  -  Menu  structure  too  deep  and  non-intuitive. 

CDU  -  Pilot  flying  can’t  not  set  altimeter  setting  or  not  see  what  it  is. 

CDU  -  Too  many  button  pushes  to  set  altimeter  setting. 
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•  COM/NAV/IFF  -  Workload  too  high.  Also  a  heads-down  operation.  Several  button 
pushes  to  get  to  a  OM  or  NAV  preset  page.  Workload  intolerable,  bit  too  high  for  simple 
tasks. 

•  Could  not  remember  how  (where  to  go)  to  set  new  altimeter  settings.  Just  confirming 
altimeter  settings  required  2  button  pushes. 

•  Could  not  see  HSD  indications  with  DIGMAP  (digital  mapping  system)  running  in  the 
background.  White  on  white  does  not  work. 

•  Determining  Com/Nav  presets  requires  several  button  pushes.  Appears  to  take  less  time 
if  I  “fat  finger”  them. 

•  DIGMAP  -  Disappeared  a  couple  of  times.  No  overlay  of  flight  plan  if  displayed  on  top 
(1/2  page). 

•  DIGMAP  -  Menu  and  overlay  structure  confusing.  Configured  by  trial  and  error. 

•  Digmap  -  No  aircraft  symbol  if  on  top  half  of  display. 

•  DIGMAP  -  Overlays  and  scaling  is  confusing.  EICAS  needs  one  all-inclusive  page. 

•  DIGMAP  -  The  overlays  don’t  have  enough  contrast  between  the  overlays  and  the  map. 

•  DIGMAP  -  With  the  jog  map  selected,  the  HSD  compass  ring  and  flight  plan  depiction 
were  hard  to  distinguish  on  the  map.  The  white  lines  were  hard  to  see.  It  would  be  nicer 
to  have  a  shadowed  line  or  a  different  color. 

•  DIGMAP  -  Zoom  feature  is  buried  too  deep  in  menus.  Could  not  find  immediately. 
Zoom  ratio  is  only  displayed  on  one  overlay.  Should  always  be  displayed. 

•  DMS  -  Problem  occurred  due  to  my  lack  of  familiarality  with  the  system.  Many  menus 
and  it  is  difficult  to  work  the  zoom  feature. 

•  EICAS  -  Can’t  put  all  functions  where  you  want.  Fuel  should  always  be  displayed  on 
EICAS  or  HSD.  Torque  was  redundant  and  not  necessary. 

•  EICAS  -  Don’t  need  torque.  Could  not  be  out  in  the  upper  left  MFD  even  though  an 
HSD  was  up  in  the  upper  right  MFD.  Add  fuel  to  the  EICAS  page  (burn  rate  and  total). 

•  EICAS  -  Engine  instruments  and  caution/advisory  on  separate  pages. 

•  EICAS  -  Engine/transmission  bar  scale  page  needs  updating!  Need  fuel  bar/window,  Ng 
bar!  Must  be  able  to  put  this  page  where  you  want  it  (e.g.,  inside  top  MFD). 

•  EICAS  -  Need  one  well  designed  EICAS  with  engine  and  transmission  instruments, 
caution/advisory,  and  fuel  on  one  page. 

•  EICAS  -  Same  as  WCA  notes.  Also,  the  EICAS  only  displays  digital  Ng.  This  is  not 
good  enough.  Should  have  Ng  tape  to  aid  in  quick  readability  of  Ng  trend  info. 

•  EICAS  -  Too  many  displays  to  see  what  I  need.  Needs  to  be  one  page. 

•  EICAS  -  Too  many  pages  to  display  the  infonnation  needed  (e.g.,  1  page  should  be  able 
to  show  all  systems  and  fuel).  Designed  properly,  1  page  can  show  all  fuel,  systems,  and 
cautions/advisories. 

•  EICAS  -  Too  many  pages  to  see  all  that  is  required  for  flight.  Need  fuel  bars,  Ng  bars  on 
engine/transmission  bar  graph  page,  and  page  must  be  able  to  be  put  in  a  top  window  of 
MFD. 

•  EICAS  -  Torque  meter  is  redundant.  Replace  with  total  fuel  and  burn  rate. 

•  EICAS  -  Unable  to  monitor  entire  aircraft  condition  on  1  page. 

•  EICAS  needs  to  be  able  to  move  to  upper  left  (inside)  MFD.  Also,  needs  to  have  fuel  bar 
on  it,  with  a  digit  fuel  flow  and  amount  window.  Get  rid  of  TQ  bar  -  it’s  already  on  the 
VSD.  Move  TQ  req/avail  to  bottom  of  TQ  bar  on  VSD. 
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•  FLT  Plan  -  I  could  not  figure  out  quickly  how  to  go  “direct-to”  to  a  preset  ACP  point.  I 
had  to  use  the  MFCU. 

•  GPS/FLT  Plan  -  Again,  kind  of  workload  intensive.  The  workload  rating  is  not 
unbearable,  just  too  high  for  some  of  the  required  tasks.  Also  primary  “heads-down” 
operation. 

•  GPS/FP  -  Too  many  button  pushes  to  get  things  accomplished.  LSK  labels  not  intuitive 
-  must  know  where  something  is  in  order  to  select  it. 

•  HSD  -  ACP,  ADF,  VOR,  TACAN  needles  should  be  different  colors.  Need  heading  bug 
without  flight  director. 

•  HSD  -  CDI  was  not  directional  when  flying  outbound  on  VOR  course. 

•  HSD  -  Compass  rose  and  bearing  pointers  difficult  to  read  with  light  colored 
background.  Some  map  backgrounds  useless  with  compass  rose.  No  reference  bug 
available  for  use  on  compass  rose.  NAV  pointers  all  need  to  be  color  coded. 

•  HSD  -  Compass  rose  and  CDI  pointers  very  hard  to  read  with  light  colored  background 
maps.  Compass  rose  needs  four  letter  cardinal  headings  depicted  on  compass  rose  (N,  S, 
E,  W). 

•  HSD  -  Configuring  the  NAV  information  on  the  HSD  is  confusing.  There  are  too  many 
choices  (CDI  has  at  least  3  choices)  to  make  a  quick  accurate  selection  at  this  level  of 
proficiency. 

•  HSD  -  Could  not  bring  up  heading  bug  without  engaging  flight  director. 

•  HSD  -  Could  not  determine  how  to  display  the  flight  plan  (ACP)  route  overlayed  on  the 
HSD  to  see  my  route. 

•  HSD  -  Could  not  get  VOR  or  ADF  big  pointers  to  come  up.  All  bearing  pointers  being 
the  same  color  is  hard  to  distinguish  between  them.  I  believe  that  FD  and  overlays 
should  be  combined  into  one  page.  I  would  like  to  see  the  MFK  control  the  last  thing  I 
touch  on  the  MFD  bezel. 

•  HSD  -  If  you  have  a  VOR  bearing  pointer  and  ACP  bearing  pointer  both  on  and  they 
point  to  the  same  location,  it  is  hard  to  tell  if  one  or  the  other  or  both  are  on.  The  needle 
head  displays  both  “D”  and  “V”. 

•  HSD  -  Need  easy  way  of  setting  hdg  and  crs  bugs. 

•  HSD  -  Numbers  along  the  compass  ring  were  too  small  to  distinguish  quickly. 

•  HSD  -  Selecting  bearing  pointer  modes  and  CDI  modes  is  confusing  on  your  own  side, 
not  to  mention  trying  to  determine  what  the  other  pilot  has  set. 

•  HSD  -  The  CDI  and  compass  pose  too  hard  to  read  with  light  background  maps.  Could 
not  see  CDI  with  Jog  up 

•  HSDH  -  The  corrective  action  cues  were  not  intuitive.  Could  not  maintain  position  using 
cues. 

•  HSDH  -  The  hover  page  displayed  the  donut  90  degrees  off  to  the  right  of  actual  aircraft 
movement.  Also  found  it  unnatural  to  be  trying  to  focus  inside  on  a  hover  page  when 
trying  to  hover.  Especially  with  my  hover  page  on  lower  half  of  outbound  MFD. 

•  I  would  like  to  see  more  lines  below  50%  on  the  TQ  strips.  When  perfonning  the  HIT 
check,  I  usually  use  the  strips  to  get  close,  then  digits  to  get  accurate. 

•  IFF  -  Change  mode  3A  should  be  a  top-level  function. 

•  IFF  -  Setting  mode  3A  is  too  deep  into  the  menus. 

•  Manage  COM/NAV/IFF  -  Paths  to  radios  and  Navaids  are  not  intuitive.  Too  much 
scrolling  and  not  enough  one  touch  hot  keys. 
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•  Manage  flight  plan  -  Could  not  get  flight  plan  overlay  on  left  HIS. 

•  MAP  -  Very  difficult  to  find  the  zoom  feature.  Not  intuitive. 

•  MFD  -  With  CDI(VOR)  up  -  when  selecting  FD  and  then  go  back  to  Other  pages,  CDI 
goes  off. 

•  Roll  command  and  pitch  command  indicators  are  prominent  on  the  VSD.  I  kept  trying  to 
fly  them  back  to  the  wings  instead  of  the  reverse.  If  they  could  be  shown  in  the 
background  -  behind  the  wings,  I  believe  it  would  have  been  better  for  me. 

•  TDL  -  Using  MFD  it  was  too  long.  Too  many  screens  to  go  through.  The  net  I  was 
sending  on  when  using  the  MFD  did  not  link  to  the  radio  I  had  set  and  displayed  on  the 
MFD.  I  had  to  reset  each  time.  It  should  remember  the  net  I  had  last. 

•  The  velocity  vector  on  the  VSDH  was  90  degrees  off.  The  doughnut  was  also  inaccurate. 

•  VSD  -  After  changing  airspeed  value  for  Flight  Director,  my  airspeed  bug  remained  on 
old  value  until  separately  loaded  into  my  CDU. 

•  VSD  -  Angle  of  bank  indicator  opposite.  No  Kollsman  window. 

•  VSD  -  Flight  director  altitude  mode  was  incorrect. 

•  VSD  -  Had  difficulty  identifying  a  slight  bank  of  the  aircraft  and  with  the  turn  indicator 
displaying  opposite  of  the  turn  direction,  corrected  to  the  wrong  direction  frequently. 

•  VSD  -  Need  fuel,  Kollsman  window,  and  Nr,  Ng,  and  TGT  pop-ups 

•  VSD  -  Need  to  combine  FD  and  overlay.  PFD  needs  fuel,  Nr,  Kollsman  window,  TGT, 
and  Ng  pop-ups. 

•  VSD  -  Processed  data  was  confusing  in  a  20  degree  and  greater  angle  of  hank. 
Maintaining  airspeed  using  only  processed  data  was  impossible. 

•  VSD  -  Roll  indicator  indicates  wrong  direction  for  roll  out.  Bar  scales  for  a/s  and  alt,  but 
round  type  gages  would  be  easier  and  quicker  to  use.  GS  readout  needs  to  be  below  a/s 
tape.  No  altimeter  window  caused  pilot  to  not  know  what  was  set. 

•  VSD  -  Same  as  my  previous  comments  +  fuel  and  altimeter  setting  needs  to  be 
displayed. 

•  VSD  -  Setting  FD  modes  and  values  on  the  VSD  is  complicated.  Must  use  the  MFD  and 
the  CDU  (or  CDU  and  MFK).  This  is  too  much  work  to  do  a  simple  task  such  as  a 
heading  or  airspeed  change. 

•  VSD  -  Slow  to  pick  up  on  radar  altimeter  cues  in  low  level  flight  environment 

•  VSDH  -  Hover  point  not  directional!  Should  be  able  to  move  towards  hover  point 
depiction,  not  away! 

•  VSDH  -  Velocity  vector  and  acceleration  cues  were  not  working. 

•  WCA  -  Could  not  acknowledge  the  main  transmission  chip. 

•  WCA  -  Need  to  integrate  caution/advisory  onto  one  well  designed  EICAS. 

•  WCA  -  Too  many  pages.  A  pop-up  WCA  summary  page,  well  designed,  is  all  that  is 
necessary.  Too  many  page  selections  to  get  the  infonnation. 

•  When  sending  position  reports,  the  frequency  should  remain  constant  until  changed  by 
the  crewmember.  Having  to  select  A2C2S  and  439  each  time  added  9  button  pushes  to 
the  1 1  it  takes  to  send  a  position  report. 
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PV2  Comments 


Rate  the  ease  of  use  for  the  VSD/HSD  screens  on 
the  MFD. 
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Rate  the  ease  of  use  for  the  DMS. 
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Rate  the  ease  of  use  of  the  CDU. 


NOTE:  6%  of  the  responses  indicated  they  did  not  use  this  piece 
of  equipment  during  the  mission. 
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Rate  the  ease  of  use  of  the  EICAS. 
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Rate  the  ease  of  use  of  the  TDL  messages. 


Rate  the  ease  of  use  of  the  flight  director. 
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NOTE:  17%  of  the  responses  indicated  they  did  not  use  the  flight 
directorduring  the  mission. 


•  Barometric  altimeter  setting  page  is  not  intuitive. 

•  CDU  -  Awkward  and  time  consuming  to  use  with  the  MFK.  Going  from  MFK  to  CDU 
and  then  to  MFK  again  takes  too  long  and  will  not  be  used  effectively.  There  must  be 
more  hot  keys  on  CDU  (e.g.,  altimeter  hot  key,  CDI  hot  key,  transponder  hot  key)  or 
additional  space  with  hot  keys. 
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•  CDU  -  Data  entry  for  the  pilot  on  the  controls  is  too  difficult  for  the  basic  data  entry 
functions  (e.g.,  make  the  MFD  slaved  to  hot  key  functions  only  such  as  altimeter, 
heading,  course  set  display. 

•  CDU  -  LSK  labeling  not  intuitive.  Some  “top-level”  functions  (e.g.,  altimeter  buried  too 
deep). 

•  CDU  -  Menu  intensive.  Too  many  layers.  Many  functions  that  should  be  “top-level”  are 
2  or  3  levels  down  (e.g.,  altimeter  setting,  selecting  a  new  heading,  etc.).  Some 
keys/locations  not  intuitive  (e.g.,  IDX,  PRO/P ATT/ACPs). 

•  CDU  -  Menus  too  deep  and  not  intuitive  (i.e.,  altimeter  setting,  direct-to,  ACP  locations). 

•  CDU  -  Not  intuitive  -  Menu  structure  requires  rote  memorization. 

•  CDU  -  The  limited  MFK  was  cumbersome  to  use  with  CDU  lending  data  entry  only  to 
the  CDU  keyboard.  Primary  functions  are  buried  too  deep.  Should  have  had  more  hot 
keys. 

•  CDU  -  Too  many  button  pushes  to  get  to  where  you  need  (e.g.,  setting  WP/VOR  course 

-  several  button  pushes  compared  to  a  twist  of  a  knob  on  the  A/L,  try  to  set  altimeter, 
etc). 

•  CDU  -  Very  heads-down,  non-intuitive  menus.  Setting  FD  heading,  course,  altitude,  and 
airspeed  is  very  cumbersome. 

•  Com/Nav  presets  are  buried  too  deep  to  be  effective.  I  wrote  presets  on  my  kneeboard 
prior  to  flight  to  get  around  all  the  button  pushes. 

•  Could  not  remember  where  to  put  in  new  altimeter  settings.  Should  be  on  mission  page  - 
not  hidden  under  index. 

•  DCU  -  Menus  are  not  intuitive  and  too  deep. 

•  DIGMAP  and  FD  -  Too  many  choices  and  labeling  not  logical.  I  don’t  think  additional 
display  space  is  required.  Better,  more  logical  use  of  the  space  available  is  a  better 
answer. 

•  DMS  -  All  displays  need  aircraft  symbol. 

•  DMS  -  Bezel  keys  should  show  next  page  not  current. 

•  DMS  -  Overlays  complicated  -  too  many  layers. 

•  DMS  -  Zoom  needs  to  be  relooked.  Zoom  %  should  be  on  every  page  and  is  too 
difficult.  Buried  too  deep  to  be  functional. 

•  EICAS  -  Need  one  all-inclusive  EICAS  page. 

•  EICAS  -  Need  one  well  designed  page  with  engine/transmission  instruments, 
caution/advisory,  and  fuel  on  one  page. 

•  EICAS  -  Too  many  button  pushes  to  see  all  that  needs  to  be  used  in  flight  (e.g.,  eng/trans 
instruments,  fuel). 

•  EICAS  -  Too  many  pages.  All  engine  indications,  cautionsand  advisories,  and  fuel  can 
be  put  on  one  page.  Why  have  several?  This  is  wasted  space  -  wasted  time  for  the  pilots 

-  very  inefficient. 

•  EICAS  -  Too  many  pages.  Need  one  page  with  all  systems,  caution/advisory,  and  fuel. 

•  FD  -  Need  easier  way  to  set  commanded  a/s,  alt,  hdg,  and  crs  that  is  not  so  heads  down. 

•  FD  -  Too  cumbersome  to  engage.  Too  cumbersome  to  change  values.  Too 
cumbersome! 

•  Flight  Director  -  Too  much  work  to  use  the  FD.  Also,  no  capability  for  using  the 
“heading  bug”  without  a  FD  mode  engaged. 

•  Getting  the  DIGMAP  up  on  the  top  inboard  MFD  was  difficult. 
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•  Having  to  push  HSD  to  get  out  of  the  TDL  message  was  not  intuitive. 

•  I  found  it  still  confusing  to  remember  where  items  are  located.  Best  example  today  was 
the  altimeter  setting.  I  did  not  think  “index”.  Also,  I  felt  confused  on  fast  method  of 
changing  radio  frequencies  if  given  one  from  ATC  that  was  not  a  preset. 

•  If  the  EICAS  can  be  reconfigured  and  can  be  allowed  to  move  to  any  location  on  any 
MFD,  then  5th  MFD  is  not  needed. 

•  Placing  a  preset  VOR  or  ADF  into  the  active  was  too  many  steps.  Having  to  go  through 
menus  to  presets  then  back  to  enter  the  preset  #  again  to  activate.  I  would  like  to  be  able 
to  activate  the  preset  from  the  list. 

•  TDF  -  Too  long  in  the  cockpit  going  through  screens  and  the  net  problem  previously 
mentioned. 

•  VSD  -  Must  press  too  many  buttons  -  labels  not  intuitive. 

•  VSD  off  the  map  would  not  bring  up  the  map.  Could  have  been  a  software  glitch. 

•  VSD/HSD  -  FD  too  heads  down  require  use  of  both  the  MFD  and  CDU.  Overlays  are 
confusing. 

•  VSD/HSD  -  Not  enough  information  on  each  page.  Requires  page  switching. 

•  VSD/HSD  -  Setting  up  FD  and  overlay  confusing.  Need  heading  bug. 

PV3  Comments 
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•  It  is  quicker  to  enter  data  directly  into  the  CDU.  Responds  too  slow  to  pilot  input 
(increase  sensitivity).  I  preferred  the  CDU. 

•  Recommendations  -  Link  the  MFK  to  hot  key  functions  only.  These  hot  keys  would  be 
for  critical  functions  (e.g.,  altimeter,  CDI,  transponder,  etc)  and  slave  the  MFK  to  those 
functions  only.  This  is  intuitive  and  would  decrease  pilot  workload.  The  current 
implementation  of  the  MFK  increases  pilot  workload. 

•  Use  MFK  for  altimeter  settings. 

•  By  the  time  you  go  through  the  CDU  menus  to  activate  the  MFK  for  the  intended 
purpose,  it’s  easier  to  continue  typing  the  information  into  the  CDU. 

•  Would  prefer  to  see  multiple,  single-purpose  knobs. 

•  I  don’t  think  this  is  a  useful  knob  as  it  is  now.  It  is  much  easier  to  manually  enter  the 
desired  parameter  via  the  keyboard  than  to  use  the  MFK. 

•  MFK  was  intermittent.  Should  be  able  to  use  it  to  set  or  move  a  course/heading  bug. 

•  Slave  the  MFK  to  hot  keys.  Use  only  for  primary  and  basic  functions.  If  you  have  to 
touch  the  CDU  at  all,  then  it  is  easier  to  fat  finger  the  numbers  in  on  the  CDU.  I  would 
find  it  easy  and  intuitive  to  push  an  altimeter  key  on  the  MFD  then  turn  the  MFK  to  set  it. 

•  By  the  time  you  select  the  proper  CDU  page  to  enable  the  proper  MFK  function  -  it  was 
easier  tp  just  continue  using  the  CDU  and  enter  the  info.  Implement  multiple  knobs  with 
single  functions  that  are  always  enabled. 

•  Independent  heading  bug  on  each  pilots  HSD  controlled  by  MFK. 

•  Altimeter  setting  would  also  be  a  good  use  if  barometric  setting  was  on  each  pilots  VSD. 

•  MFK  adds  to  workload.  It  is  easier  to  type  the  desired  value  in. 

•  Allow  it  to  be  used  with  a  “hot  key”  function  from  MFD.  Like  setting  altimeter,  moving 
heading  bug,  or  course. 

•  Get  rid  of  MFK  or  slew  it  to  a  specific  primary  flight  data  function.  It  is  inefficient  in  it’s 
current  capacity.  Pilots  will  use  the  CDU  input  pad.  Instead  of  buying  buttons  and  knobs 
for  a  Kollsman  window,  heading  bug,  course  set  display  -  have  one  MFK  that  works  for 
these  functions  and  maybe  a  couple  other  functions  only. 

•  Incorporate  multiple  single  use  knobs  instead  of  one  multi-use  knob.  Since  you  have  to 
enable  the  MFK’s  function  with  the  CDU  -  it’s  easier  to  keep  typing. 

•  Sim  Problem  -  I  could  not  keep  scrolling  down  -  it  seemed  like  it  got  stuck. 

•  Allow  it  to  control  what  you  touch  on  the  MFD  bezel. 
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PV4  Comments 


Did  you  encounter  any  problems  using  the 
MFCU? 
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Did  the  directional  control  of  the  MFCU  work 
as  you  expected? 
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Was  the  sensitivity  of  the  M  FCU  controller 
appropriate? 
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Not  used,  but  when  it  is  used,  it  slew  cursor  way  too  slow  and  should  be  progressive 
speed  with  pressure. 

Needed  to  be  variable  speed  with  highest  speed  faster  than  the  current  implementation. 
Having  the  bug  start  at  present  location  was  good. 

Cursor  needs  to  move  quicker. 
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PV5  Comments 


Was  there  any  sym  bology  depicted  on  the 
VSD  that  was  difficult  to  understand? 
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Was  there  any  sym  bology  depicted  on  the 
VSDH  that  was  difficult  to  understand? 


Was  there  any  sym  bology  depicted  on  the 
HSD  that  was  difficult  to  understand? 


Was  there  any  sym  bology  depicted  on  the 
HSDH  that  was  difficult  to  understand? 
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Was  there  any  sym  bology  depicted  on  the 
EICAS  that  was  difficult  to  understand? 
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Was  there  any  sym  bology  depicted  on  the 
DM  S  that  was  difficult  to  understand? 


Was  there  any  sym  bology  depicted  on  the 

TDL  that  was  difficult  to  understand? 
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•  DIG  MAP  -  White  route,  aircraft,  HSD,  HSDH  hard  to  read  overlaid  on  jog  map. 

•  Digital  Map  -  I  found  the  map  pages  and  overlays  difficult  to  find.  Make  zoom  controls 
available  every  time  a  map  is  displayed.  More  simple  map  features  may  be  better. 

•  DIGMAP  -  Finding  zoom  function  too  difficult  -  never  knew  what  scale  I  was  in. 

•  DIGMAP  -  Menu  system  and  overlays  confusing. 

•  DIGMAP  -  Overlay  menu  structure  confusing. 

•  DMS  -  Course  lines  are  very  difficult  to  see  on  a  jog.  Lines  need  to  be  shaded, 
(currently  white  lines  on  white  background). 

•  DMS  -  Need  aircraft  symbol  on  all  digmap  displays. 

•  DMS  -  Overlays  not  enough  contrast. 

•  DMS  -  Zoom  hard  to  read.  Zoom  %  should  always  be  displayed. 
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•  EICAS  -  Need  a  strip  for  Ng  -  in  my  scan,  I  could  see  the  strips  but  had  to  focus  harder 
to  see  Ng. 

•  EICAS  -  Need  one  page  with  all  the  basics  (e.g.,  add  fuel,  bars  to  RPMR,  eng,  trans 
page).  Must  be  able  to  move  to  upper  inside  MFD.  Otherwise  you  need  5th  MFD. 

•  EICAS  -  Need  one  well  designed  EICAS  page  with  all  infonnation  on  it. 

•  EICAS  -  Not  well  designed  -  should  be  one  page. 

•  EICAS  -  Too  many  button  pushed  to  see  all  that  is  needed  on  a  flight.  Ng  hard  to  read 
on  a  quick  look  in  by  pilot. 

•  EICAS  -  Too  many  button  pushes  to  see  all.  Need  to  consolidate  onto  one  good,  useable 
page. 

•  EICAS  -  Too  many  pages 

•  EICAS  -  too  many  pages. 

•  EICAS  -  Torque  is  redundant  and  should  be  better  utilized  with  fuel  and  fuel  flow.  Can’t 
put  EICAS  page  in  any  position  you  want. 

•  EICAS  -  Torque  is  redundant.  Was  not  able  to  place  screen  where  I  wanted  it. 

•  HSD  -  Cardinal  directions  are  too  small.  I  have  to  concentrate  and  focus  harder  to  see 
them. 

•  HSD  -  Compass  rose  too  light  -  can’t  read  on  light  background. 

•  HSD  -  Compass  rose/CDI  too  “white”.  Needs  to  be  darker  for  light  background  maps. 

•  HSD  -  Hard  to  see  CDI  bearing  pointer  with  light  colored  background.  Also  hard  to  see 
multiple  bearing  pointers  due  to  all  being  one  color. 

•  HSD  -  Need  different  color  for  ACP,  ADF,  VOR,  TACAN  needles. 

•  HSD  -  Numbers  on  compass  hard  to  read. 

•  HSD  -  Numbers  or  cardinal  displays  should  be  larger.  Everything  is  very  difficult  to  see 
over  a  jog  map  or  even  a  black  screen. 

•  HSD  -  Too  small  of  numbers  on  the  compass  ring.  Bearing  pointers  need  to  be  different. 

•  HSD  -  With  more  than  1  CDI  or  bearing  pointer  selected,  hard  to  tell  one  from  the  other. 
Suggest  different  sizes,  FAA/TSO  standard  colors. 

•  HSD  difficult  to  set-up  CDI/bearing  infonnation  and  difficult  to  interpret  (Different  color 
pointers  may  help  this). 

•  HSDH  -  Donut  90  degrees  to  the  right  of  the  hover  movement. 

•  Need  color  code  bearing  to  pointers.  Remove  donut  from  FD.  Need  one  all-inclusive 
EICAS  page. 

•  VSD  -  Angle  of  bank  pointer 

•  VSD  -  Angle  of  bank  pointer  going  opposite  of  aircraft  causes  disorientation. 

•  VSD  -  FD  indications  harder  to  follow  than  in  legacy  aircraft.  Roll  indicator  turns 
opposite  of  pitch  ladder  bars.  Very  disorienting. 

•  VSD  -  I  had  trouble  with  altimeter  reading.  Reported  6000  feet  instead  of  600  (actual). 

•  VSD  -  No  way  to  tell  what  altimeter  setting  is! 

•  VSD  -  Processed  data  is  not  intuitive  -  airspeed  especially  difficult  to  detennine  in  a 
bank. 

•  VSD  -  Recommend  removing  the  “doughnut”  from  the  FD. 

•  VSD  -  Turn  indicator  is  opposite  of  aircraft  bank.  This  is  confusing. 

•  VSD  -  Turn  indicator  moves  opposite  of  aircraft  hank  -  confusing.  Need  an  Nr  strip. 
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•  VSD  -  Would  like  to  have  fuel  on-board  displayed.  This  would  prevent  me  from  going 
through  another  MFD,  two  pages  deep  just  to  see  fuel  status. 

•  VSDH  -  Completely  unintuitive  -  input  commands  make  no  sense. 

•  VSDH  -  Hover  point  not  directional  (we  moved  wrong  way). 

•  VSDH  -  Not  working  correctly. 


PV6  Comments 


Rate  the  ease  or  difficulty  level  of  sending  position 
reports. 


Rate  the  ease  or  difficulty  level  of  sending  free- 
text  messages. 
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Rate  the  ease  or  difficulty  level  of  sending 
observation  reports. 
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NOTE:  83  %  of  responses  indicated  that  the  pilots  did  not  send 
observation  reports. 
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•  Did  not  have  the  time  to  send  a  free  text  message  on  the  tactical  mission.  That  would 
have  been  way  too  much  time  inside  the  cockpit. 

•  Position  Reports  -  Too  many  button  pushes  away.  The  top-level  button  to  get  to  the 
menus  is  not  intuitive.  System  makes  me  think  of  aircraft  systems  not  messages.  The  net 
#  and  Net  ID  name  did  not  match  what  we  had  active  in  the  radio.  I  had  to  reset  each 
time. 

•  On  the  MFCU,  I  did  not  see  “Position  Report”  but  it  was  easier  using  the  MFCU  to  send 
messages. 


PV7  Comments 


How  would  you  rate  your  ability  to  detect 
incoming  TDL  messages? 
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NOTE:  Riots  did  not  receive  incoming  TDL  messages. 


How  would  you  rate  your  ability  to  detect 
cautions  and  advisories? 


NOTE:  77  %  of  responses  indicated  that  the  pilots  did  not 
encounter  cautions  or  advisories. 
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How  would  you  rate  your  ability  to  detect 
warnings? 


NOTE:  88  %  of  responses  indicated  that  the  pilots  did  not 
encounter  aircraft  w  arnings. 


How  would  you  rate  your  ability  to  detect 
exceeding  operational  limits? 
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NOTE:  50  %  of  responses  indicated  that  the  pilots  did  not  exceed 
operational  limits. 


How  would  you  rate  your  ability  to  detect  low 
fuel  indications? 
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NOTE:  44  %  of  responses  indicated  that  the  pilots  did  not 
experience  low  fuel  indications. 


Low  Fuel  -  It  is  impossible  to  determine  low  fuel  unless  you  have  the  fuel  page  selected. 
Recommendations  -  Make  fuel  indications  static  on  additional  display  space  or  put  fuel 
on  a  higher  priority  page  such  as  the  HSD  page  or  the  systems  page.  Fuel  is  the  most 
limiting  factor  in  how  we  perform  Army  missions  -  so  who  exactly  chose  to  de-prioritize 
it  by  putting  it  away  and  out  of  sight.  This  needs  much  more  attention. 

There  is  no  fuel  quantity  indicator  present  unless  you  pull  up  a  big  line  drawing  and  leave 
it  up.  A  fuel  strip  should  be  present  on  the  VSD. 


81 


•  Operational  Limits  -  You  don’t  know  you’re  about  to  exceed  TGT  or  Ng  from  the  PFD. 

If  no  EICAS  page  is  selected,  exceedance  may  be  unknown.  Ng  has  no  strip  (only  digits) 
on  EICAS  page. 

•  Can  not  see  changing  trends  to  engine/transmission  systems  (bar  graphs)  or  fuel.  Must 
have  that  page  up,  too  many  button  pushes.  A  pilot  can’t  wait  until  a  system  goes  past  a 
limit. 

•  Fuel  should  never  be  out  of  sight  -  never.  Put  on  the  systems/EICAS  page  at  a  minimum. 

•  Caution/Advisory  -  Need  one  good  page  integrated  into  the  EICAS  page. 

•  Operating  Limits  -  Need  Ng,  TGT  pop-ups  on  PFD.  Need  fuel  on  PFD. 

•  I  did  not  have  the  fuel  page  up.  It  would  be  very  difficult  for  me  to  see  a  low  fuel 
condition  prior  to  the  caution  light. 

•  EICAS  -  Will  not  see  a  “trend”  without  engine/transmission  bar  graph  page  up  all  the 
time.  Will  have  to  wait  for  a  limit  to  be  exceeded  before  getting  an  alert.  Will  not  know 
fuel  is  getting  low  until  “low  Fuel”  light  comes  on.  This  is  unacceptable.  Must  be  able  to 
see  trends. 

•  Low  Fuel  -  I  want  to  see  fuel  at  all  times.  Low  fuel  warning  is  not  an  engine  or  systems 
warning.  You  can  most  of  the  time  get  the  aircraft  back  into  limits,  but  you  can’t  do  that 
with  fuel  -  so  why  use  the  same  indicating  system  for  fuel  and  systems? 

•  Poor  design  and  never  should  have  been  designed  this  way.  Big  mistake. 

•  Easy  to  see  TRQ,  but  not  other  indications  of  limits  being  reached. 

•  Would  not  know  of  low  fuel  unless  the  fuel  page  is  up  -  unless  we  were  already  in  a  low 
fuel  situation. 

•  Need  all-inclusive  EICAS.  Without  TGT  and  Ng  pop-ups,  you  can  exceed  and  never 
know  it. 

•  C/A  -  Borderline.  Only  see  chip  light.  No  indication  of  oil  pressure,  temperature,  etc. 
Unless  you  slew  through  EICAS  page. 

•  Limits  -  Hard  to  spot  trending  on  Ng.  Must  have  dedicated  EICAS  pages  up  and  rotate 
through. 

•  Fuel  -  Poor  design.  Make  one  EICAS  page. 


PV8  Comments 


•  Only  if  you  update  EICAS  page  and  be  able  to  move  it  to  inside  upper  MFD. 

•  In  the  current  configuration  with  current  problems,  you  need  another  display.  See 
previous  comments. 
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Not  without  design  changes.  I  would  want  to  see  fuel  quantity,  altimeter  setting,  and  Nr 
strip  on  VSD.  Be  able  to  place  engine  instruments  on  top  of  one  MFD.  We  don’t  need  a 
drawing  of  the  aircraft  on  a  fuel  readout.  Just  give  us  the  strips  and  numbers. 

PFD  needs  pop-ups  for  pending  engine  exceedance.  EICAS  needs  to  be  one  page,  not  4. 
Without  significant  changes,  a  5th  MFD  is  definitely  required. 

Another  MFD  was  required  for  this  mission  and  it’s  required  workload. 

Not  in  current  configuration.  If  designed  properly,  4  MFDs  can  be  adequate.  If  the 
displays  can’t  be  changed,  then  may  need  a  5a  MFD. 

5  th  MFD  is  not  required  if  EICAS  pages  can  be  changed. 

Need  more  space  to  display  fuel,  Kollsman  window,  and  hot  keys  or  fix  the  issues 
addressed. 

Need  to  redesign  PFD  and  EICAS  pages. 

Not  in  its  current  available  page  format  addition  of  info  (i.e.,  comments  we  discussed  in 
AARs  this  week  need  to  be  incorporated  or  else  a  5th  MFD  needs  to  be  added.  If  we  can’t 
change  page  format  the  5th  MFD  would  be  better  than  just  4. 

I  don’t  think  a  5th  MFD,  if  it  has  the  exact  same  displays,  will  alleviate  the  problem.  You 
cannot  get  a  VSD,  HSD,  Systems,  and  WCA  and  Fuel  on  2  MFD’s.  This  is 
unsatisfactory  and  a  big  problem  with  this  design.  Putting  an  identical  5  th  MFD  is  not  a 
solution  -  it  is  a  reaction  to  a  poor  design. 

But  software  needs  to  be  updates  for  what  is  displayed  and  where. 

In  current  configuration  with  current  software,  you  must  have  a  5th  MFD. 

Don’t  add  a  5th  MFD.  Fix  the  ones  we  have. 

Not  in  current  CAAS  page  format.  Both  the  PFD  and  EICAS  need  additions  discussed  in 
AAR’s.  EICAS  must  be  one  catch-all  page  showing  all.  If  fixes  are  incorporated  into 
CAAS  pages  I  think  5  MFD’s  are  not  needed. 

I  think  a  total  redesign  is  called  for  or  use  displays  from  UH-60M  baseline. 
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Do  you  feel  that  the  training  you  received  in 
the  CAAS  cockpit  was  adequate  to  perform 
this  m ission? 
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•  Training  was  adequate.  Proficiency  will  be  an  issue.  Pilots  will  have  to  use  the  system 
daily  to  remember  how  to  get  to  everything.  Finding  things  is  not  always  intuitive. 

•  More  time  to  fly  the  pitch  and  roll  command  indicators.  Deleting  the  doughnut  from 
inside  the  CMD  indicators  may  also  help. 
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Intentionally  left  blank 
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Glossary  of  Acronyms 


AAR 

after-action  review 

ACP 

air  control  point 

AMRDEC 

Aviation  and  Missile  Research,  Development,  and  Engineering  Center 

AOI 

area  of  interest 

APEX 

Advanced  Prototyping,  Engineering,  and  experimentation  (Laboratory) 

ARL 

Army  Research  Laboratory 

ASL 

Applied  Science  Laboratories 

ATM 

aircrew  training  manual 

ATTC 

Aviation  Technical  Test  Center 

BHIVE 

Battlefield  Highly  Immersive  Virtual  Environment 

BWRS 

Bedford  Workload  Rating  Scale 

CAAS 

Common  Avionics  Architecture  System 

CDU 

control  display  unit 

CPC 

Comanche  portable  cockpit 

DIGMAP 

digital  mapping  system 

EDS 

engineering  development  simulator 

EICAS 

engine  instrument  caution  advisory  system 

EUD 

early  user  demonstration 

HFE 

human  factors  engineering 

HSD 

horizontal  situation  display 

IFR 

instrument  flight  rules 

IIMC 

inadvertent  instrument  meteorological  conditions 

IMC 

instrumented  meteorological  conditions 

LEUE 

limited  early  user  evaluation 

LUT 

limited  user  test 

MANPRINT 

manpower  and  personnel  integration 

MFCU 

multi-function  control  unit 

MFK 

multi-function  knob 

MEDEVAC 

medical  evacuation 

MFD 

multi-function  display 

OTW 

out  the  window 

PFD 

primary  flight  display 

PO 

Product  Office 

PVI 

pilot-vehicle  interface 

SA 

situational  awareness 

SART 

Situational  Awareness  Rating  Technique 

SSDD 

Systems  Simulation  and  Development  Directorate 

SSQ 

Simulator  Sickness  Questionnaire 

TDL 

tactical  data  link 

TRADOC 

Training  and  Doctrine  Command 

TSC 

tactical  steering  committee 

TSM 

TRADOC  System  Manager 

UH 

utility  helicopter 

VFR 

visual  flight  rules 
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VMC 

VSD 

WSRT 

XP 


visual  meteorological  conditions 
vertical  situation  display 
Wilcoxon  Signed  Ranks  Test 
experimental  test  pilot 
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NO.  OF 

COPIES  ORGANIZATION 

1  DEFENSE  TECHNICAL 
(PDF  INFORMATION  CTR 
ONLY)  DTICOCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BELVOIR  VA  22060-6218 

1  US  ARMY  RSRCH  DEV  &  ENGRG  CMD 
SYSTEMS  OF  SYSTEMS 
INTEGRATION 
AMSRD  SS  T 
6000  6TH  ST  STE  100 
FORT  BELVOIR  VA  22060-5608 

1  INST  FOR  ADVNCD  TCHNLGY 
THE  UNIV  OF  TEXAS  AT  AUSTIN 
3925  W  BRAKER  LN  STE  400 
AUSTIN  TX  78759-5316 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMNE  ALC  IMS 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
AMSRD  ARL  Cl  OK  TL 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

2  DIRECTOR 

US  ARMY  RESEARCH  LAB 
AMSRD  ARL  CS  OK  T 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

UNIT  OF  ACTION  MANEUVER  BATTLE  LAB 
ATTN  ATZKUA 
BLDG  1101 

FORT  KNOX  KY  40121 

1  ARLHRED  AVNC  FLD  ELMT 

ATTN  AMSRD  ARL  HR  MJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FT  RUCKER  AL  36362-5000 

1  ARLHRED  AMCOM  AUN  FLD  ELMT 
ATTN  AMSRD  ARL  HR  MI 
BLDG  5464  RM  202  J  MINNINGER 
REDSTONE  ARSENAL  AL  35898-7290 


NO.  OF 

COPIES  ORGANIZATION 

1  ARLHRED  AMCOM  MSL  FLD  ELMT 
ATTN  AMSRD  ARL  HR  MO  T  COOK 
BLDG  5400  RM  C242 
REDSTONE  ARS  AL  35898-7290 

1  ARL  HRED  USAADASCH  FLD  ELMT 
ATTN  AMSRD  ARL  HR  ME  A  MARES 
ATTN  ATSACD 
5800  CARTER  ROAD 
FORT  BLISS  TX  79916-3802 

1  ARLHRED  ARDEC  FLD  ELMT 

ATTN  AMSRD  ARL  HR  MG  R  SPINE 
BUILDING  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARL  HRED  ARMC  FLD  ELMT 

ATTN  AMSRD  ARL  HR  MH  C  BURNS 
BLDG  1467B  ROOM  336 
THIRD  AVENUE 
FT  KNOX  KY  40121 

1  ARLHRED  CECOM  FLD  ELMT 

ATTN  AMSRD  ARL  HR  ML  J  MARTIN 
MYER  CENTER  RM  2D3 1 1 
FT  MONMOUTH  NJ  07703-5630 

1  ARL  HRED  FT  BELVOIR  FLD  ELMT 

ATTN  AMSRD  ARL  HR  MK  J  REINHART 

10125  KINGMAN  RD 

FORT  BELVOIR  VA  22060-5828 

1  ARLHRED  FT  HOOD  FLD  ELMT 

ATTN  AMSRD  ARL  HR  S  MIDDLEBROOKS 
MV  HQ  USAOTC 
91012  STATION  AVE  ROOM  111 
FT  HOOD  TX  76544-5073 

1  ARL  HRED  FT  HUACHUCA  FLD  ELMT 
ATTN  AMSRD  ARL  HR  MY  M  BARNES 
2520  HEALY  AVE  STE  1172  BLDG  51005 
FT  HUACHUCA  AZ  85613-7069 

1  ARL  HRED  FLW  FLD  ELMT 

ATTN  AMSRD  ARL  HR  MZ  A  DAVISON 
320  MANSCEN  LOOP  STE  166 
FT  LEONARD  WOOD  MO  65473-8929 

1  ARLHRED  NATICK  FLD  ELMT 

ATTN  AMSRD  ARL  HR  MQ  M  R  FLETCHER 
NATICK  SOLDIER  CTR  BLDG  3 
AMSRD  ARL  NSC  SE  E 
NATICK  MA  01760-5020 
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NO.  OF 
COPIES 

1 

1 

1 

1 

1 

1 

1 


1 


ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


ARL  HRED  SC&FG  FLD  ELMT 
ATTN  AMSRD  ARL  HR  MS  C  MANASCO 
SIGNAL  TOWERS  RM  3  03 A 
FORT  GORDON  GA  30905-5233 

ARL  HRED  STRICOM  FLD  ELMT 
ATTN  AMSRD  ARL  HR  MT  C  CHEN 
12350  RESEARCH  PARKWAY 
ORLANDO  FL  32826-3276 

ARL  HRED  TACOM  FLD  ELMT 
ATTN  AMSRD  ARL  HR  MU  M  SINGAPORE 
6501  E  11  MILE  RD  MAIL  STOP  284 
BLDG  200A  2ND  FL  RM  2104 
WARREN  MI  48397-5000 

ARL  HRED  USAFAS  FLD  ELMT 
ATTN  AMSRD  ARL  HR  MF  C  HERNANDEZ 
BLDG  3040  RM  220 
FORT  SILL  OK  73503-5600 

ARL  HRED  USAIC  FLD  ELMT 
ATTN  AMSRD  ARL  HR  MW  E  REDDEN 
BLDG  4  ROOM  332 
FT  BENNING  GA  31905-5400 

ARL  HRED  USASOC  FLD  ELMT 
ATTN  AMSRD  ARL  HR  MN  R  SPENCER 
DCSFDI  HF 

HQ  USASOC  BLDG  E2929 
FORT  BRAGG  NC  28310-5000 

ARL  HRED  FT  LEAVENWORTH  FLD  ELMT 
ATTN  AMSRD  ARL  HR  MP 
D  UNGVARSKY 
BATTLE  CMD  BATTLE  LAB 
415  SHERMAN  AVE  UNIT  3 
FT  LEAVENWORTH  KS  66027-2326 

ARL  HRED  AMEDD  FLD  ELMT 
ATTN  AMSRD  ARL  HR  MM  V  RICE 
BLDG  4011  RM  217 
1750  GREELEY  RD 
FT  SAM  HOUSTON  TX  78234-5094 


1  ARL  HRED  JFCOM  FLD  ELMT 

ATTN  AMSRD  ARL  HR  MJF  D  BARNETTE 
JFCOM  JOINT  EXPERIMENTATION  J9 
JOINT  FUTURES  LAB 
115  LAKEVIEW  PKWY  STE  B 
SUFFOLK  VA  23535 

1  US  ARMY  SAFETY  CTR 

ATTN  CSSC  SE 
FORT  RUCKER  AL  36362 

1  MICRO  ANALYSIS  &  DESIGN  INC 
ATTN  BETHPLOTT 
4949  PEARL  E  CR  #300 
BOULDER  CO  80301 

ABERDEEN  PROVING  GROUND 

1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 
ATTN  AMSRD  ARL  Cl  OK  TECH  LIB 
BLDG  4600 

1  US  ATEC 

RYAN  BLDG 
APG-AA 

1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 

ATTN  AMSRD  ARL  Cl  OK  TP  S  FOPPIANO 

BLDG  459 

1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 
ATTN  AMSRD  ARL  HR  MB  J  NAWLEY 
BLDG  459 

1  DIRECTOR 

US  ARMY  RSCH  LABORATORY 
ATTN  AMSRD  ARL  HR  M  F  PARAGALLO 
BLDG  459 


ARL  HRED  SPO 

ATTN  AMSRD  ARL  HR  M  M  STRUB 
6359  WALKER  LAND  STE  100 
ALEXANDRIA  VA  22310 
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